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ABSTRACT 


This  study  evaluates  the  usability  of  a  U.S.  Navy 
Decision  Support  System  (DSS) .  The  DSS  was  developed  to 
enhance  the  performance  of  tactical  decision-makers 
within  a  Navy  Combat  Information  Center.  The  goals  of 
this  study  were  to  test  the  DSS  against  usability 
criteria  and  objectives  to  track  future  redesign  efforts 
and  system  improvements.  The  purpose  of  this  analysis 
was  to  (1)  assess  the  system's  usability,  (2)  identify 
problems  areas  in  the  graphical  user  interface,  (3) 
report  trends  in  user  feedback,  and  (4)  provide 

recommendations  addressing  major  usability  issues 
encountered  by  participants.  The  study  tested  whether 
the  DSS  met  the  usability  objectives  of  (a)  90% 

successful  task  completion,  (b)  ease-of-use  ratings  of 
somewhat  easy  or  better,  and  (c)  satisfaction  ratings  of 
somewhat  satisfied  or  better.  The  DSS  did  not  meet  these 
usability  objectives  for  task  completion  or  ease-or-use, 
however  the  DSS  did  meet  the  usability  objective  for 
user  satisfaction.  All  participants  reported  that  they 
enjoyed  working  with  the  DSS  and  believed  that  it  would 
be  a  significant  step  forward  in  information  management. 
Based  on  the  usability  data  gathered  in  the  study, 
recommendations  are  provided  to  address  the  usability 
issues. 
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EXECUTIVE  SUMMARY 


This  study  evaluates  the  usability  of  a  U.S.  Navy 
Decision  Support  System  (DSS) .  The  DSS  was  developed  to 
enhance  the  performance  of  tactical  decision-makers 
within  a  Navy  Combat  Information  Center.  The  DSS  is 
still  in  the  development  phase  and  has  continually  been 
improved  based  on  empirical  studies  and  subject  matter 
expertise.  The  most  recent  prototype  version,  known  as 
the  DSS-2,  is  the  focus  of  this  study.  The  goals  of  this 
study  were  to  test  the  DSS-2  against  usability  criteria 
and  objectives  to  track  future  redesign  efforts  and 
system  improvements.  The  purpose  of  this  analysis  was  to 
(1)  assess  the  system's  usability,  (2)  identify  problems 
areas  in  the  graphical  user  interface,  (3)  report  trends 
in  user  feedback,  and  (4)  provide  recommendations 
addressing  major  usability  issues  encountered  by 
participants.  The  study  tested  whether  the  DSS  met  the 
usability  objectives  of  (a)  90%  successful  task 
completion,  (b)  ease-of-use  ratings  of  somewhat  easy  or 
better,  and  (c)  satisfaction  ratings  of  somewhat 
satisfied  or  better.  The  DSS-2  did  not  meet  these 
usability  objectives  for  task  completion  or  ease-or-use, 
however  the  DSS-2  did  meet  the  usability  objective  for 


XV 


user  satisfaction.  All  participants  reported  that  they 
enjoyed  working  with  the  DSS-2  and  believed  that  it 
would  be  a  significant  step  forward  in  CIC  information 
management.  Based  on  the  usability  data  gathered  in  the 
study,  recommendations  are  provided  to  address  the 
usability  issues. 

The  methodology  applied  in  this  study  was  useful  in 
the  evaluation  of  the  DSS-2.  This  study  demonstrated 
that  traditional  human -computer  interface  usability 
methods  could  be  directly  applied  the  evaluation  of 
synthetic  environments.  The  DSS-2  is  a  simple  synthetic 
environment  represented  on  two  computer  monitors.  Given 
the  success  of  this  methodology  with  the  DSS-2,  it  would 
be  appropriate  to  use  this  methodology  in  evaluating 
more  complex  synthetic  environments. 
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I. 


INTRODUCTION 


A.  OVERVIEW 

The  United  States  Navy  and  Marine  Corps  strategy  of 
forward  presence  suggests  that  they  will  be  first  on  the 
scene  in  times  of  crisis.  Furthermore,  since  a  majority 
of  the  world’s  population  lives  within  200  miles  of  the 
ocean,  most  future  contingencies  are  likely  to  involve 
littoral  warfare  (Mundy,  1994) .  This  strategy  will 
increasingly  place  Naval  forces  in  coastal  areas  where 
they  will  be  forced  to  operate  in  confined  and  congested 
areas  (Hutchins,  Kelly,  &  Morrison,  1997)  .  These 
constraints  will  result  in  operational  scenarios  that 
will  require  both  increased  information  processing  and 
accelerated  decision-making.  These  challenges  are 
exacerbated  by  the  fact  that  current  real-time  battle 
management  systems  are  primarily  effective  in  dealing 
with  all-out  conflicts  and  not  particularly  capable  in 
situations,  such  as  littoral  conflicts,  where  human 
intervention  in  decision-making  is  more  critical 
(Hutchins,  Morrison,  and  Kelley,  1996). 

Two  unfortunate  and  well-known  incidents  involving 
the  U.S.S.  Stark  and  the  U.S.S.  Vincennes  highlight  this 
challenge.  The  U.S.S  Stark  incident  centers  on  a 
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decision  made  by  the  commander  not  to  engage  an  inbound 
aircraft.  The  aircraft  was  not  considered  a  threat  by 
the  commander,  however  it  was  a  threat.  The  aircraft 
significantly  damaged  the  U.S.S.  Stark  and  numerous 
lives  on  board  were  lost.  The  commander  of  the  U.S.S. 
Vincennes  faced  a  similar  problem,  yet  believed  the 
inbound  aircraft  he  faced  was  a  threat  to  his  ship.  As  a 
result,  the  inbound  aircraft  was  destroyed  by  the  U.S.S. 
Vincennes.  The  aircraft  turned  out  to  be  a  commercial 
passenger  airline  and  all  lives  onboard  the  airliner 
were  lost.  The  Tactical  Decision-Making  Under  Stress 
(TADMUS)  program  was  initiated  to  address  these  types  of 
problems.  The  principle  product  of  the  TADMUS  program  is 
the  Decision  Support  System  (DSS) .  Due  to  the 
criticality  of  these  issues  and  the  need  to  correctly 
identify  threats,  it  is  imperative  that  the  design  of 
the  TADMUS  DSS  system  be  intuitive  and  easy  to  use.  This 
study  will  evaluate  the  usability  of  the  TADMUS  DSS 
graphical  user  interface. 

B.  BACKGROUND 

The  DSS  system  was  developed  to  enhance  the 
performance  of  tactical  decision-makers.  It  was  derived 
from  current  cognitive  theory.  This  derivation  first 
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analyzed  the  cognitive  tasks  performed  by  decision 
makers  in  a  shipboard  Combat  Information  Center  and 
second,  developed  a  set  of  displays  to  support  these 
tasks  based  on  the  underlying  decision  making  processes 
(Morrison  et.  al .  ,  1997).  The  DSS  is  currently  a 
prototype  and  is  planned  to  be  formally  tested  onboard  a 
Navy  ship  in  1999,  see  Figure  1. 


Figure  1:  TADMUS  DSS  Integrated  Display 


The  DSS  is  still  in  the  development  phase  and  has 
continually  been  improved  based  on  empirical  studies  and 
subject  matter  expertise.  The  most  recent  prototype 
version,  known  as  the  DSS-2,  is  the  focus  of  this  study. 
A  usability  test  was  conducted  on  the  DSS  system  to 
evaluate  human  performance  and  user  preferences.  This 
test  also  identifies  usability  issues  that  focus  on 
future  design  and  redesign  efforts. 
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C.  PROBLEM  STATEMENT 


Given  the  critical  nature  of  the  tasks  supported  by 
the  DSS  system,  and  the  implications  of  a  difficult  to 
use  design,  a  usability  study  was  conducted  on  the  DSS. 
The  goals  of  this  study  were: 

•  Conduct  a  usability  study  to  test  where  the  DSS-2 
product  is  today  in  regards  to  usability 
criteria . 

•  Track  usability  measures  (e.g.,  successful 
completion  of  tasks,  error  rate,  time  to  complete 
task,  ease-of-use,  and  user  satisfaction)  in 
order  to  track  product  improvement. 

•  Identify  usability  issues  to  address  future 
design  iterations. 

•  Provide  user  feedback  to  DSS  development  team. 

•  Provide  recommendations  to  address  usability 
issues  encountered  by  users  during  testing. 

D .  OBJECTIVE 

The  purpose  of  this  study  is  to  assess  the 
usability  of  the  human -computer  interface  of  the  TADMUS 
DSS.  The  long-term  objective  is  to  provide  a  methodology 
and  baseline  information  for  the  evaluation  of  future 
systems . 
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E.  SCOPE  AND  LIMITATIONS 


To  narrow  the 
computer  interaction 
analyzed.  Issues 
decision-making  will 


scope  of  the  thesis,  only  human- 
performance  and  preference  will  be 
concerning  conceptual  cognitive 
not  be  addressed. 
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II.  LITERATURE  REVIEW 


A.  OVERVIEW 

The  review  of  literature  for  this  research  included 
journals  and  textbooks  covering  the  subjects  of 
usability  evaluation,  human-computer  interaction,  and 
synthetic  environments.  The  purpose  of  this  literature 
review  is  to  provide  an  overview  of  the  historic  and 
current  theories  and  practices  relating  to  usability 
evaluation  and  to  provide  information  on  the  methods 
used  in  this  study  to  evaluate  the  DSS. 

B .  BACKGROUND 

There  has  been  significant  growth  in  the  fields  of 
synthetic  environments  and  usability  engineering.  The 
term  synthetic  environment  is  used  to  refer  to  virtual 
reality,  virtual  environments,  teleoperator  system, 
telerobotic  systems,  augmented  reality  and  synthetic 
environments  in  general.  However,  these  two  fields  have 
experienced  growth  independently.  "An  underlying 
assumption  among  both  (synthetic  environments) 
researchers  and  developers  sometimes  seems  to  be  that 
(synthetic  environments) ,  because  they  are  a  novel  and 
impressive  technology,  are  inherently  good  and  usable 
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(Gabbard  and  Hix,  1997,  p.3)."  The  tools  developed  and 
the  lessons  learned  in  the  field  of  usability- 
engineering  have  yet  to  be  significantly  applied  to 
synthetic  environments  and  those  that  have  been  applied 
typically  have  not  addressed  the  broad  issues  of 
usability  throughout  the  system  (National  Research 
Council,  1997;  and  Gabbard  and  Hix,  1997).  The 
integration  of  these  two  fields  will  mutually  benefit 
both.  Usability  engineering  will  gain  a  technologically 
savvy  customer  and  developers  of  synthetic  environments 
will  drastically  improve  the  usability  of  their 
technologically  complex,  and  frequently  difficult  to  use 
systems . 

Usability  engineers  will  need  to  modify  existing 
methods  and  tools  as  well  as  develop  new  ones 
specifically  for  synthetic  environments.  For  example, 
typical  human-computer  interaction  usability  studies 
focus  on  standard  graphical  user  interfaces  where  there 
is  a  single  user.  In  the  synthetic  environment, 
innovative  and  non-standard  methods  of  interaction  as 
well  as  a  multi-user  capability  call  for  a  redefinition 
of  the  current  usability  paradigm.  To  facilitate  this 
transition,  Gabbard  and  Hix  (1997)  have  outlined  the 


8 


four  primary  usability  characteristics  related  to 
synthetic  environments.  These  are: 

1.  Users  and  User  Tasks  in  Synthetic  Environments  - 
general  user  and  task  characteristics  and  types 
of  tasks  in  synthetic  environments. 

2 .  The  Virtual  Model  -  usability  characteristics  of 
generic  components  typically  found  in  synthetic 
environments . 

3 .  Synthetic  Environment  User  Interface  Input 

Mechanisms  -  usability  characteristics  of 

synthetic  environment  input  devices . 

4.  Synthetic  Environment  User  Interface  Presentation 

Components  -  usability  characteristics  of 

synthetic  environment  output  devices . 

Gabbard  and  Hix  have  developed  a  comprehensive 
taxonomy  based  on  these  four  areas  in  order  to  move 
beyond  the  "let's  build  it  and  see  what  happens"  method 
that  is  often  employed  in  synthetic  environments.  This 
taxonomy  is  a  classification,  enumeration,  and 

discussion  of  usability  issues  in  synthetic  environments 
and  was  developed  to  ensure  that  usability  will  be 
integrated  into  the  development  of  synthetic 
environments.  In  addition  to  this  work,  additional 
analysis  of  usability  engineering  and  its  integration 
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into  synthetic  environments  through  usability  testing 
needs  to  be  undertaken.  The  first  step  in  determining 
how  this  can  be  accomplished  is  to  examine  usability, 
usability  engineering,  and  synthetic  environments. 

C .  USABILITY 

Usability  engineering  is  a  systematic  approach  to 
usability.  In  general,  usability  means  that  the  people 
who  use  the  product  are  able  to  do  so  quickly  and  easily 
to  accomplish  their  own  tasks  (Dumas  and  Redish,  1994) . 
This  definition  is  based  on  four  essential  points,  they 
are : 

1 .  Focus  on  users . 

2  .  People  utilize  products  to  be  productive. 

3  .  People  have  limited  time  to  accomplish  tasks. 

4  .  Users  decide  when  a  product  is  easy  to  use. 
Usability  is  concerned  with  the  sum  total  of  a  product. 
Usability  should  not  only  be  considered  an  issue  for  the 
primary  system  functionality,  but  should  also  be  applied 
to  training  materials,  help  packages,  and  other 
associated  features  of  the  system.  In  order  to  improve 
the  ease-of-use  of  a  product,  usability  should  be 
considered  throughout  the  development  of  a  system,  from 
initial  design  through  final  deployment  of  the  system. 
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Dumas  and  Redish  (1994)  provide  seven  principles 
for  ensuring  usability: 

1.  Engineering  it  into  a  product  through  an  iterative 
design  and  development  process. 

2 .  Involving  users  throughout  the  process . 

3 .  Allowing  usability  and  users '  needs  to  drive  design 
decisions. 

4 .  Working  in  teams  that  include  skilled  usability 
specialists,  interface  designers,  and  technical 
communicators . 

5.  Setting  quantitative  usability  goals  early  in  the 
process . 

6.  Testing  products  for  usability,  but  also  integrating 
usability  testing  with  other  methods  for  ensuring 
usability. 

7 .  Being  committed  to  making  technology  work  for 
people. 

This  integration  of  usability  into  a  product  is  commonly 
called  usability  engineering,  (Good,  1988;  Whiteside, 
Bennett,  and  Holtzblatt,  1987)  .  Similar  to  software 
engineering,  usability  engineering  includes  identifying 
users,  analyzing  tasks,  setting  specifications, 
developing  and  testing  prototypes,  and  the  iterative 
cycles  of  development  and  testing  (Dumas  and  Redish, 
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1994) .  Gould  and  Lewis  (1985)  highlight  four  principles 
to  facilitate  designing  usability  into  products. 

1 .  Focus  early  and  continuously  on  users. 

2.  Integrate  consideration  of  all  aspects  of 
usability. 

3  .  Test  versions  with  users  early  and  continuously. 

4.  Iterate  the  design. 

Identifying  usability  requirements  prior  to  design  can 
save  time  and  money  for  the  designer  as  well  as  increase 
the  likelihood  of  user  satisfaction  with  the  product. 
Systems  are  developed  to  help  individuals  accomplish  a 
task.  In  order  to  provide  a  usable  system,  what  the 
individual  needs  and  how  they  are  to  accomplish  this 
must  be  ascertained.  The  primary  requirement  is  to 
understand  the  prospective  users  and  the  audience  for  a 
system.  Dumas  and  Redish  (1994)  have  identified 
techniques  that  can  be  used  in  a  usability  engineering 
process.  These  techniques  highlight  the  importance  of 
describing  what  a  person  does  in  their  job  in  terms  of 
tasks.  When  the  tasks  are  analyzed,  how  the  person  does 
the  job,  can  do  the  job,  or  should  do  the  job  are 
described  (Drury,  Paramore,  Van  Cott,  Grey,  and  Corlett, 
1987)  . 
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Table  1:  Usability  Engineering  Process  Techniques 

Techni(?ues  for  Planning,  Implementing,  and  Evaluating  Usability 
Uncovering  usability  needs  before  you  design 
Identifying  users'  jobs  and  tasks 
Convening  focus  groups 

Interviewing  and  observing  users  in  context 
Conducting  usability  tests  of  existing  versions 
Conducting  usability  tests  of  competitors'  products 
Setting  quantitative  usability  goals 
Basing  designs  on  expertise  in  human- computer  interaction  (HCI) 
Understanding  the  HCI  and  document  design  approach 
Using  HCI  and  document  design  principles  and  guidelines 
Setting  and  using  local  rules 
Evaluating  usability  throughout  design  and  development 
Getting  experts  to  review  the  design 
Having  peers  or  experts  walk  through  the  design 
Having  users  work  with  static  prototypes 
Having  users  work  with  interactive  prototypes 
Getting  user  edits  on  early  versions  of  documentation 
Conducting  iterative  usability  tests 

_ Asking  users  about  their  satisfaction _ _ 

Redish  and  Dumas  (1994) 


In  addition  to  understanding  the  principles  of 
usability  and  the  usability  engineering  process,  it  is 
important  to  set  quantifiable  usability  goals  early  in 
the  design  process.  By  setting  quantifiable  goals,  a 
product  development  team  will  have  a  concrete  way  to 
measure  usability  success.  A  series  of  quantitative 
goals  with  related  objectives  should  be  identified  prior 
to  system  development  and  will  facilitate  subsequent 
analysis.  A  team  may  have  a  goal  to  design  a  product  to 
be  easy  to  learn  and  operate,  however  this  is  not  a 
quantitative  goal  and  would  be  difficult  to  measure.  The 
design  team  needs  to  define  quantitative  goals  to  more 
easily  measure  usability.  Subjective  criteria  can  also 
be  defined  to  help  evaluate  a  products'  usability. 
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Subjective  criteria  are  often  easily  derived,  yet  it  can 
be  difficult  to  determine  if  the  criteria  have  been  met. 
Typically,  it  is  easier  to  determine  when  objective 
goals  have  been  met.  However,  they  may  originally  be 
more  difficult  than  the  subjective  criteria  to  develop. 
For  instance,  in  an  air  traffic-control  synthetic 
environment,  a  usability  goal  might  be  that  users  should 
be  able  to  detect  and  identify  a  new  track  in  less  than 
5  seconds.  Measures  such  as  these  are  important  in  that 
they  provide  a  basis  for  evaluation  on  whether  the  goal 
was  achieved,  they  allow  systems  to  be  compared,  and 
provide  baseline  information  against  which  revisions  can 
be  evaluated. 

D.  USER  INTERFACE  DESIGN 

The  usability  of  a  product  is  inherently  tied  to 
the  user  interface.  If  the  user  interface  is  intuitive, 
easy  to  learn  and  use,  a  product  will  have  favorable 
usability  ratings.  Guidelines  and  user  interface 
heuristics  have  been  established  by  academia  and 
industry  experts  to  best  design  user  interfaces  for 
usability.  Shneiderman  (1997)  proposes  eight  golden 
rules  of  interface  design  to  best  maximize  the  usability 
of  an  interface.  These  include: 
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1 .  strive  for  consistency 

2 .  enable  frequent  users  to  use  shortcuts 

3 .  offer  informative  feedback 

4.  design  dialogs  to  yield  closure 

5.  offer  error  prevention  and  simple  error  handling 

6.  permit  easy  reversal  of  actions 

7 .  support  internal  locus  of  control 

8.  reduce  short-term  memory  load 

If  followed,  these  rules  should  foster  a  sense  of 
comprehension  and  competence  among  users.  This  is 
particularly  important  because  users  prefer  systems  with 
which  they  feel  familiar  and  competent.  Furthermore,  if 
a  user  has  positive  feelings  toward  a  system  they  are 
more  likely  to  highly  rate  the  performance  of  these 
systems.  These  rules  were  originally  developed  primarily 
for  the  standard  graphical  user  interface.  However, 
these  general  underlying  principles  of  interface  design 
can  be  interpreted,  refined,  and  extended  to  synthetic 
environments . 

Striving  for  consistency  can  be  problematic  in  that 
consistency  can  relate  to  many  aspects  of  the  system 
(i.e.,  terminology,  color,  layout,  input  and  display 
formats) .  For  example,  consistency  in  a  virtual 
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walkthrough  of  a  house  could  refer  to  consistency  of  the 
visual  representation  of  objects  in  the  environment  or 
could  refer  to  consistency  of  human  interaction  with 
these  features.  It  is  not  always  possible  to  maintain 
consistency  across  all  dimensions  of  a  system,  but 
identical  symbology  and  methods  of  interaction  should  be 
employed  throughout . 

Shortcuts  enable  frequent  users  to  reduce  the 
number  of  interactions  required  to  obtain  a  desired 
result  and  also  increase  the  pace  of  interaction.  In  the 
synthetic  environment,  unique  methods  of  input  and 
display  need  to  be  improved  to  take  better  advantage  of 
shortcuts.  For  instance,  there  may  be  times  when  a 
three-dimensional  virtual  environment  could  be  switched 
to  a  two-dimensional  map  on  which  the  user  could 
navigate.  When  the  user  reaches  a  desired  location,  a 
return  to  the  three-dimensional  world  could  be 
initiated.  Other  shortcuts  could  include  gestures  or  a 
series  of  gestures,  which  are  not  directly  relevant  in 
the  current  environment,  but  offer  a  shortcut  to  another 
environment . 

Offering  information  feedback  facilitates  the 
user’s  immersion  in  synthetic  environments.  This 
feedback  can  vary  in  degree  with  infrequent  and  minor 
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actions  resulting  in  small  changes  in  the  synthetic 
environment,  whereas  critical  and  major  actions  result 
in  substantial  changes  in  the  visual  presentation. 
Without  substantial  information  feedback  users  may  not 
be  able  to  fully  complete  actions  or  understand  their 
current  status.  For  example,  in  an  air-traffic  control 
system  the  selection  of  an  aircraft  should  be  indicated 
through  a  state  change  (i.e.,  the  display  alters  and  an 
object  becomes  highlighted) . 

Usability  of  a  system  can  be  further  maximized  by 
designing  dialogs  to  yield  closure.  This  can  be  achieved 
by  grouping  a  set  of  actions  to  provide  a  natural  flow 
through  a  users'  tasks.  This  sequencing  of  actions 
provides  the  user  better  awareness  of  the  actions  taken 
and  gives  the  user  a  sense  of  closure  of  the  sequence. 
An  example  of  this  concept  is  virtual  kitchens  where  a 
user  can  pick  up  a  dish,  manipulate  the  dish,  and  break 
the  dish.  If  correctly  employed,  the  user  will  clearly 
know  the  status  of  the  dish  and  the  associated  action. 
At  the  conclusion  of  the  sequence  of  actions,  the  user 
will  clearly  see  the  dish  replaced  on  a  counter  or 
broken  into  several  pieces  and  be  rewarded  through  this 
sense  of  closure  and  awareness. 
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whenever  practical,  users  should  be  permitted  to 
reverse  actions  if  they  choose.  Users  tend  to  make 
mistakes,  therefore  a  system  should  be  designed  to  allow 
users  to  recover  from  errors  easily.  This  may  reduce  any 
stress  or  anxiety  the  user  has  when  operating  within  a 
synthetic  environment.  In  the  virtual  kitchen  example 
mentioned  above,  a  user  may  have  selected  the  "wrong" 
dish.  The  user  should  be  able  to  recover  from  this  error 
and  replace  the  dish  in  the  same  place  as  before. 
However,  the  virtual  kitchen  example  also  suggests  when 
this  would  potentially  not  be  allowed.  For  example,  if 
the  user  has  broken  a  dish,  there  is  no  recreation  of. 
the  dish.  Similarly,  if  in  a  missile  fire-control 
station  a  missile  is  accidentally  fired  there  is  no 
ability  to  call  it  home. 

The  design  of  a  system  should  also  support  a  users' 
"internal  locus  of  control."  Users  should  be  the 
initiators  of  actions  not  the  responders  to  actions 
(Gaines,  1981).  When  appropriate,  a  synthetic 
environment  should  be  designed  with  the  users  in 
command.  Whereas  the  status  of  objects  in  the 
environment  would  be  appropriately  updated  and 
maintained  without  user  action,  the  autonomous  movement 
of  the  user  within  the  environment  or  a  drastic  altering 
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of  the  visual  orientation  would  be  inappropriate.  The 
issue  of  system  interruptions  arises  in  this  context 
(McFarlane,  1998) .  If  a  user  is  engaged  in  the  synthetic 
environment,  when  is  it  appropriate  for  the  user  to  be 
interrupted  with  a  competing  task?  How  should  that 
interruption  manifest  itself?  This  issue  of 
interruptions  needs  further  study,  in  particular  within 
the  context  of  virtual  environments . 

The  reduction  of  short-term  memory  load  is 
essential  for  optimal  integration  of  a  user  within  a 
synthetic  environment.  Humans  are  limited  in  their 
ability  to  maintain  excessive  amounts  of  information  in 
their  short-term  memory.  Designs  of  synthetic 
environments  should  include  cues,  mnemonics,  and 
standardized  sequences  of  actions.  Whenever  possible, 
access  to  integrated  assistance  information  should  be 
provided.  For  instance,  in  an  air  traffic-control 
environment,  the  history  of  the  air  tracks  should  be 
made  available  to  the  user.  If  a  task  requires  a  series 
of  actions,  a  list  of  those  actions  should  be  available. 
When  designing  systems  to  reduce  short-term  memory  load, 
the  designer  should  remember  that  humans  have  been  shown 
to  be  able  to  remember  seven  items  plus  or  minus  two. 
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These  rules  of  interface  design,  which  are  based 
upon  existing  usability  models,  can  be  modified  for  the 
synthetic  environment.  However,  the  limitations  of 
existing  usability  models  should  be  understood. 
Synthetic  environments  have  many  unique  characteristics 
and  understanding  these  is  essential  in  addressing  the 
overall  usability.  Unique  characteristics  of  synthetic 
environments  not  supported  in  existing  usability  models 
include  perceived  presence  and  perceived  real  world 
fidelity  and  existing  models  do  not  support 
quantification  or  qualification  of  a  user's  perception 
of  such  characteristics  (Gabbard  and  Hix,  1997). 
Traditional  usability  models  are  also  limited  in  scope 
in  that  they  typically  focus  on  a  single  user  at  a 
single  site.  Other  issues  involve  the  multiple  and 
unique  methods  of  interaction  and  display  that  are 
continually  being  developed  for  synthetic  environments. 

E.  USABILITY  TESTING 

A  usability  test  primarily  measures  ease-of-use. 
According  to  Dumas  and  Redish  (1994),  "usability  testing 
is  a  systematic  way  of  observing  actual  users  trying  out 
a  product  and  collecting  information  about  the  specific 
ways  in  which  the  product  is  easy  or  difficult  for  them 
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to  use  (p.  12)."  Dumas  and  Redish  (1994)  also  identify 
three  basic  tenets  of  usability  testing.  The  first  is 
that  usability  testing  should  be  used  to  diagnose 
problems  and  not  to  determine  that  the  product  is 
flawless.  The  second  is  that  usability  testing  should  be 
employed  early  in  the  development  of  a  product  and 
often.  Lastly,  that  usability  testing  is  part  of  a 
process  that  focuses  on  usability  throughout  design  and 
development . 

In  order  to  best  incorporate  usability  into  the 
development  process,  a  thorough  testing  plan  needs  to  be 
developed.  There  are  several  determinants  that  need  to 
be  addressed  in  developing  an  evaluation  plan 
(Shneiderman,  1997;  Nielsen,  1993;  Hix  and  Hartson, 
1993;  Preece  et  al .  ,  1994;  Newman  and  Lamming,  1995).  A 
foundational  determinant  is  the  current  stage  of  the 
design.  The  requirements  for  testing  an  early  design  as 
compared  to  a  late  design  will  differ  in  that  general 
concepts  of  user  interaction  with  the  design  need  to  be 
tested  early,  whereas  testing  of  a  late  design  may  be 
targeted  more  at  identifying  consistency  within  the 
environment  and  task  completion.  In  addition,  the 
criticality  of  the  environment  is  a  significant 
determinant  in  deciding  the  objectives  of  the  test.  The 
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level  of  task  completion  rates  and  number  of  errors 
allowed  in  a  test  will  vary  depending  on  whether  an 
environment  is  being  developed  as  part  of  a  life- 
critical  system  or  as  an  entertainment  system.  Finally, 
factors  such  as  the  novelty  of  the  project,  the  number 
of  expected  users,  the  time  available,  the  costs  of  the 
product,  the  available  resources  (i.e.,  time  and  money 
available  for  testing) ,  and  the  experience  of  the 
usability  testers  themselves  play  a  role  in  shaping  the 
usability  test. 

Usability  testing  of  a  system  has  become  essential 
not  only  to  maximize  the  usability  of  the  system,  but 
also  to  verify  that  contractual  requirements  have  been 
met  and  to  document  that  testing  has  been  conducted  in 
case  legal  issues  or  lawsuits  arise  concerning  the 
operating  of  the  system  ( Shneiderman,  1997) .  This  is 
best  illustrated  when  one  considers  that  perfection  is 
not  possible  in  any  system,  particularly  systems  that 
incorporate  human  users.  The  varying  degrees  for  which 
errors  will  be  tolerated  relates  directly  to  the 
requirements  to  bring  the  system  to  full  operational  use 
and  the  impact  that  the  errors  may  have  during 
operational  use.  However  as  Shneiderman  (1997)  suggests, 
systems  which  require  high  levels  of  input  such  as 
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nuclear-reactor-control  or  air-traffic-control 
emergencies  are  very  difficult  to  test.  However,  testing 
methods  to  deal  with  stressful  situations,  which  include 
life-critical  applications,  are  increasingly  needed. 

Another  usability  testing  method  employed  to 
improve  a  product's  usability  is  an  expert  evaluation  of 
the  system.  Nielsen  and  Mack  (1994)  argue  that  formal 
expert  reviews  can  generally  provide  more  useful 
information  as  compared  to  informal  demonstrations  to 
colleagues  or  customers.  This  requires  that  expert 
reviewers  are  available  to  the  usability  testing  team. 
If  available,  expert  reviewers  can  be  employed 
throughout  the  design  and  testing  of  a  system.  The 
typical  product  of  an  expert  review  is  a  report 
outlining  identified  problems  and  recommendations  for 
improvement.  The  forms  these  reviews  may  take  include 
heuristic  evaluation,  guideline  review,  consistency 
inspection,  cognitive-walkthrough,  and  formal  usability 
inspection.  Expert-reviews  do  face  challenges.  For 
instance,  expert-reviewers  may  be  confronted  with  new 
systems  and  technology  they  are  not  completely  familiar 
with  and  for  which  they  may  not  fully  understand  the 
design  rationale  or  development  history.  However,  expert 
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reviews  typically  provide  a  fresh  look  at  a  system  and 


are  useful  in  evaluating  system  development. 


Table  2 :  Expert  Review  Methodology 


Expert -Review 
_ Method 

Heuristic 

evaluation 


Guidelines 

review 

Consistency 

inspection 


Cognitive- 

walkthrough 


Formal 

usability 

inspection 


Description 


Expert-reviewers  critique  an  interface  to 
determine  confoimiance  with  a  short  list  of 
design  heuristics  such  as  the  eight  golden 
rules . 

The  interface  is  checked  for  conformance 
with  the  organizational  or  other  guidelines 
document 

Experts  verify  consistency  across  a  family 
of  interfaces,  checking  for  consistency  of 
terminology,  color,  layout,  input  and 
output  formats,  within  the  interfaces  as 
well  as  in  the  training  materials  and 
online  help 

Experts  simulate  users  walking  through  the 
interface  to  carry  out  typical  tasks . 
Simulating  the  day  in  the  life  of  the  user 
should  be  part  of  the  evaluation. 

Experts  hold  courtroom-style  meeting,  with 
a  moderator  to  judge,  to  present  the 
interface  and  to  discuss  its  merits  and 
weakness . 


Shneiderman,  1997 


Formal  usability  testing  in  laboratories  can 
provide  information  concerning  user  needs  and  abilities 
that  an  expert-review  may  miss.  Usability  testing  and 
usability  laboratories  have  been  developed  to  capture 
the  user  experience  directly.  The  information  gained  is 
used  to  confirm  progress  in  the  design  of  a  system  and 
to  obtain  recommendations  to  improve  upon  the  system. 
Typically,  a  formal  usability  study  is  conducted  in  a 
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usability  laboratory  in  a  controlled  setting  with  a  set 
of  tasks  for  the  user  to  undertake . 

Usability  studies  do  take  other  forms,  such  as 
Nielsen's  (1993)  discount  usability  engineering  which 
are  "quick  and  dirty"  approaches  to  task  analysis, 
prototyping,  and  testing  (Shneiderman,  1997).  Field 
studies  are  another  type  of  usability  study  which  are 
conducted  in  actual  work  environments  in  order  to 
achieve  realistic,  user  evaluation.  A  different  approach 
to  these  traditional  methods  is  to  challenge  actual 
users  of  the  system  to  try  to  break  the  system,  commonly 
called  beta  testing.  By  offering  rewards  to  individuals 
who  find  flaws  in  a  system,  developers  can  speed  up  the 
development  process  and  correct  errors  that  may  have 
been  missed  through  conventional  testing.  Two  serious 
flaws  with  usability  testing  in  general  are  that  it 
emphasizes  first-time  usage  and  lacks  a  comprehensive 
evaluation  of  the  system  due  to  time  constraints 
(Shneiderman,  1997) .  These  flaws  necessitate  that 
usability  testing  be  supplemented  with  other  methods  of 
evaluation  such  as  expert-review. 
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F.  PLANNING  FOR  USABILITY  TESTING 


When  planning  a  usability  test,  often  the  most 
important  question,  besides  what  is  required,  is  how 
long  the  test  should  take.  If  the  usability  testing  is 
an  integrated  part  of  the  design  process  and  is  not 
simply  being  conducted  on  a  completed  system,  then  the 
test  needs  to  be  as  short  as  possible  to  obtain  the 
necessary  information  -  and  short  enough  so  that  the 
test  is  not  burdensome.  This  will  facilitate  the 
iterative  nature  of  proper  usability  testing.  Testing 
length  depends  on  many  factors,  including  how  much  prior 
testing  has  taken  place,  how  complex  the  system  is,  and 
the  scope  of  the  system  to  be  tested.  Dumas  and  Redish 
(1994)  suggest  that  traditional  testing  lengths  fall 
into  one  of  four  categories.  Organizations  that  follow 
formal  testing  and  generate  comprehensive  test  reports 
allow  eight  to  twelve  weeks.  Shortened  testing  periods 
of  four  to  six  weeks  are  frequently  used  when  there 
exists  a  strong  collaboration  between  team  members  and  a 
shortened  formal  report  is  used.  When  only  a  particular 
aspect  of  a  system  is  to  be  studied  with  well- 
established  procedures,  one  week  can  suffice.  Just-in- 
time  testing  is  discouraged,  but  can  still  provide 
useful  information  in  a  couple  of  days  if  necessary. 


26 


Studies  conducted  by  experienced  and  dedicated 
individuals  who  take  the  necessary  time  will  most  often 
achieve  the  best  results.  Furthermore,  proper  planning 
entails  the  definition  of  goals  and  concerns,  deciding 
who  should  participate  and  recruiting  these  individuals, 
developing  and  organizing  tasks  and  task  scenarios, 
deciding  on  usability  measures,  preparing  the  test 
materials  and  test  environment,  and  conducting  a  pilot 
test . 


Dumas  and  Reddish  (1994)  suggest  that  defining 

goals  and  concerns  for  usability  testing  can  be  viewed 

as  a  three-stage  process.  The  first  stage  is  making 

choices  among  goals  and  concerns.  For  instance. 

Is  your  main  concern  whether  new  users 
will  be  able  to  get  up  and  running  to 
do  basic  tasks  quickly,  or  whether 
users  who  have  had  the  product  for  6 
months  can  figure  out  more  advanced 
functions?  You  may  be  concerned  about 
both,  but  you'll  have  to  plan  two 
different  tests  to  learn  about  both 
(Dumas  and  Redish,  1994,  p.  Ill)  . 

The  second  stage  is  moving  from  general  concerns  to 

specific  ones.  This  helps  determine  the  type  of  subjects 

necessary  and  begins  to  shape  concerns  into  quantitative 

objectives.  Lastly,  understanding  the  sources  of  these 

goals  and  concerns  allows  the  usability  engineer  to 

better  develop  the  testing  scenarios  and  tasks.  Some 
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sources  include  expert-reviews,  user  feedback,  and 
previous  tests. 

The  decision  on  who  should  participate  in  the  study 
should  be  based  on  developed  user  profiles.  Ideally, 
user  profiles  should  have  been  developed  prior  to  design 
of  the  system  and  usability  testing.  If  this  has  not 
been  done,  a  user  profile  can  be  developed  by 
identifying  all  the  relevant  characteristics  that  an 
individual  using  the  system  should  have.  The  two  primary 
characteristics  of  concern  are  those  that  all  the 
individuals  have  in  common  and  those  that  may  make  a 
difference  between  the  individuals.  For  example,  if  a 
command  and  control  synthetic  environment  were  being 
deployed  onboard  a  US  Navy  ship  for  the  first  time, 
certain  questions  need  to  be  addressed,  these  include: 

1. Will  many  users  be  working  with  abstract  or 
simulated  environments  for  the  first  time? 

2 .  Will  many  individuals  be  experienced  with 
personal  computer  applications,  but  new  to  the 
synthetic  environment? 

3.  Will  many  users  already  be  adept  at  using  the 
input  devices? 

4 .  Who  will  be  using  this  system  --  commanding 
officers,  junior  officers,  or  enlisted  personnel? 
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The  decision  on  how  many  subjects  are  required  for 
a  study  can  also  be  a  challenging  question.  Usability 
engineers  are  often  delighted  to  have  ten  to  twelve 
subjects  participate  in  a  study,  whereas  a  statistician 
might  insist  on  no  less  than  thirty-six  to  forty-eight. 
The  realities  of  time  and  budget  constraints  often 
result  in  usability  studies  having  six  to  eight 
subjects.  A  simple  answer  to  the  question  of  how  many 
subjects  to  use  is  enough  participants  to  complete  the 
study  as  efficiently  as  possible. 

A  method  of  determining  subject  size  proposed  by 
Bailey  (1997)  is  based  upon  a  variation  of  the  binomial 
probability  formula, 

1  -  (  1  -  P  )“ 

where, 

p  =  probability  of  the  event  occurring 
n  =  number  of  test  subjects 

For  this  method,  a  subject  matter  expert  or  team  of 
experts  must  first  derive  the  likelihood,  an  estimate 
for  p,  that  an  element  of  the  system  will  confuse  any 
one  test  subject.  For  example,  suppose  that  it  has  been 
determined  by  a  panel  of  experts  that  the  likelihood  of 
any  one  test  subject  having  difficulty  identifying  a 
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confusing  air-track  icon  in  a  synthetic  environment  is 
0.5,  that  is  to  say  using  a  single  subject  in  our  study 


there  is  a  50/50  chance  of  the  subject  having  a  problem. 
If  two  subjects  are  tested,  the  probability  that  the 
confusing  air-track  icon  be  identified  rises  to  0.75. 
This  probability  is  calculated  using  the  binomial 
formula  presented.  Furthermore,  if  three  subjects  were 
used,  this  probability  rises  to  0.87  and  with  seven 
subjects  to  0.99.  Therefore,  by  basing  our  calculations 
on  the  original  likelihood,  we  can  determine  sample 
size.  Table  3  provides  a  chart  for  determining  sample 
size.  Problem  probability  can  be  roughly  assumed  to  be  a 
rough  estimate  of  problem  severity  (Bailey,  1997)  . 


Table  3:  Likelihood  of  Performance  Test  Subjects  Having 

Problems 


Problem 

Prob6d>ility 

Number 

of 

Test 

Subjects 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

.05 

.05 

.10 

.14 

.19 

.23 

.26 

-.31 

.34 

.37 

.41 

.10 

.10 

.19 

.27 

.34 

.41 

.47 

.53 

.57 

.61 

.65 

.15 

.15 

.28 

.39 

.48 

.56 

.62 

.68 

.73 

.77 

.80 

.25 

.25 

.44 

.58 

.68 

.76 

.82 

.87 

.90 

.92 

.94 

.50 

.50 

.75 

.87 

.94 

.97 

.98 

.99 

.75 

.75 

.94 

.98 

.99 

.90 

.90 

.99 

Bailey,  1997 


After  determining  the  number  of  participants  and 
the  length  of  the  study,  the  test  can  be  developed.  Once 
the  goals  and  concerns  of  the  test  have  been  defined, 
the  initial  tasks  to  test  should  be  selected  and 
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organized  to  best  address  these  goals  and  issues.  The 
tasks  then  need  to  be  placed  in  a  context  that  is 
understandable  to  the  user.  It  is  through  the 
development  of  task  scenarios  that  this  is  accomplished. 
These  scenarios  serve  as  the  basis  for  the  test. 

6.  CONCLUSIONS 

Synthetic  environments  are  an  emerging  technology 
that  will  enable  individuals  to  perform  new  functions 
and  accomplish  older  functions  in  a  new  way.  Usability 
engineering  and  usability  testing  tools  have  recently 
been  developed  primarily  in  conjunction  with  standard 
graphical  user  interfaces.  If  synthetic  environments  are 
going  to  mature  and  become  integrated  into  our  society, 
they  must  be  easy  to  use  and  enable  individuals  to 
accomplish  tasks  more  efficiently.  This  maturation  can 
only  take  place  if  these  fields  can  be  integrated  so  as 
to  establish  systematic  and  standardized  methods  of 
evaluation.  This  evolution  will  begin  by  basing 
synthetic  environment  designs  on  human-computer 
interaction  principles. 

This  thesis  argues  that  the  application  of 
principles  of  human -computer  interaction  derived  from 
existing  literature  and  research  can  be  applied  to 
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synthetic  environments.  There  is  significant  enthusiasm 
surrounding  synthetic  environments,  but  little  effort 
has  been  made  to  mature  the  field  of  synthetic 
environments  and  utilize  existing  models  of  usability 
and  user  interface  design.  This  can  be  best  addressed  by 
demonstrating  what  is  accomplished  when  the  methods  of 
usability  are  applied  to  synthetic  environments.  It  will 
only  be  through  a  significant  development  and  successful 
implementation  of  an  existing  or  emerging  synthetic 
environment  using  usability  principles  throughout  its 
design  and  implementation  that  this  integration  will  be 
taken  seriously. 
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III. 


METHODOLOGY 


A.  RESEARCH  APPROACH 

This  study  involved  the  analysis  of  an  existing 
decision  support  system  and  the  development  of 
evaluation  methods  based  on  this  system.  The  purpose  of 
this  analysis  was  to  assess  the  extent  of  the  system’s 
usability,  to  assess  the  effect  of  the  interface  on  the 
user,  and  to  identify  any  specific  problems  with  the 
-^system  (Dix  et  al,  1997). 

B.  DATA  COLLECTION 

Participants .  12  participants  for  this  study  were 

recruited  at  the  Naval  Postgraduate  School  (NPS)  in 
Monterey,  California.  All  participants  were  military 
officer  instructors  or  students  at  NPS  and  had 
previously  served  as  Surface  Warfare  Officers  (SWO) .  Six 
of  the  participants  had  served  aboard  Aegis  ships  and 
six  had  not.  The  participants  were  further  divided  by 
experience  level  into  one  of  two  categories,  low  and 
high.  Experience  levels  were  based  on  a  combination  of 
months  spent  at  sea  and  the  number  of  deployments.  The 
low  experience  level  group  on  average  had  44  months  at 
sea  and  two  deployments;  the  high  experience  group  on 
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average  had  5  8  months  at  sea  and  an  average  of  2.7 


deployments.  Participants  were  distributed  across  four 
categories.  Table  4  shows  the  distribution  of 
participants  across  these  categories. 


Table  4:  Subject  Distribution 


Low 

High 

Experience 

experience 

Non-Aegis 

3 

3 

Aegis 

3 

3 

All  participants  had  experience  as  Combat  Information 
Center  (CIC)  Watch  Officers.  Five  participants  had 
additional  experience  as  Tactical  Action  Officers.  Two 
of  the  subjects  were  U.S.  Navy  Lieutenant  Commanders  and 
ten  were  Lieutenants. 

Instrument .  This  study  will  provide  a  benchmark 
across  usability  objectives.  A  usability  task  script  and 
post-task  questionnaire  were  administered  to  all 
subjects.  At  the  conclusion  of  the  study,  a  post-test 
questionnaire  was  administered  to  the  participants.  See 
the  descriptions  below  for  specific  definitions  of 
obj  ectives . 

•  90%  Successful  completion  of  tasks. 

•  90%  Error  free  rate. 
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•  90%  score  of  3  or  better  on  a  7  pt.  scale  (e.g.. 


l=easy,  3=somewhat  easy,  5=somewhat  difficult, 
and  7=difficult)  in  ease-of-use. 

•  90%  score  of  3  or  better  on  a  7  pt.  scale  (e.g., 

l=satisfied,  3=somewhat  satisfied,  5=somewhat 
dissatisfied,  and  7=dissatisf ied)  in  user 
satisfaction . 

Ideally,  by  the  time  a  Decision  Support  System  is 
released  to  the  fleet,  these  objectives  should  be  met 
and/or  exceeded  in  order  for  the  system  to  meet  high 
ease-of-use  standards. 

Procedure :  Participants  completed  an  informed 

consent  form  and  demographic  questionnaire  (Appendix  A 
and  B) .  The  participants  also  received  a  usability  task 
script  along  with  a  brief  description  of  the  evaluation 
scenario  (Appendix  C) .  Participants  sat  directly  in 
front  of  two  21-inch  computer  display  monitors  and 
controlled  the  DSS-2  with  a  computer  mouse.  The 
beginning  of  the  usability  evaluation  consisted  of  the 
participants  responding  to  a  series  of  questions 
concerning  their  initial  reaction  to  the  DSS-2  graphical 
user  interface  (Appendix  D) .  Participants  were  then 
directed  to  read  aloud  and  execute  the  tasks  provided 
them  in  the  task  script.  Following  each  series  of  tasks. 
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questions  concerning  the  usability  of  the  DSS-2  were 
presented.  Additional  questions  concerning  participant 
satisfaction  as  well  as  current  understanding  of  the 
DSS-2  were  also  presented.  Upon  completing  this  phase  of 
the  study,  participants  were  timed  on  the  completion  of 
tasks  using  the  DSS-2 .  The  study  concluded  with  the 
administration  of  a  post-test  questionnaire. 
Participants  received  no  training  on  the  DSS-2  prior  to 
the  usability  study.  The  DSS-2  component  names,  such  as 
track  profile  and  response  manager,  were  not  used  during 
interactions  with  the  participants.  These  components 
were  addressed  in  respect  to  the  location  they  would  be 
found  on  the  display  monitor.  For  example,  the  track 
profile  component  would  be  referred  to  as  the  area  in 
the  upper  left  side  of  the  right  monitor. 

Throughout  each  usability  session,  the  following 
measurements  were  taken  during  the  performance  of  user 
tasks.  These  measurements  were  used  to  assess  whether  or 
not  each  usability  objective  had  been  met.  These 
measurements  include: 

•  Task  Completion  Rate:  The  proportion  of  participants 
who  complete  the  task  successfully  and  independently 
without  critical  errors.  A  critical  error  has 
occurred  when  the  participant  either  requests 
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assistance  from  the  usability  engineer  or  commits  an 
uncorrected  error  that  results  in  an  incorrect 
outcome  for  the  task. 

•  Error  Free  Rate:  The  proportion  of  participants 
completing  the  task  without  any  errors,  critical  or 
non-critical .  Non-critical  errors  include  any  error 
corrected  by  the  test  participant  without 
intervention  by  the  usability  engineer  or  an  error 
left  uncorrected,  but  which  does  not  affect  the 
correctness  of  the  outcome  of  the  task. 

•  User  Satisfaction:  The  User  Satisfaction  rating  is  . 
derived  from  a  series  of  questions  which  the  user 
rates  on  a  7 -point  scale,  ranging  from  very 
dissatisfied  to  very  satisfied.  The  questions 
solicit  user  opinions  with  regard  to  ease-of-use, 
simplicity  of  the  human- computer  interaction,  system 
functionality,  and  general  satisfaction  with  the 
product . 

C.  DATA  ANALYSIS 

The  occurrence  of  each  of  the  measurements  listed 
above  was  recorded  in  a  spreadsheet.  These  data  included 
any  associated  user-feedback  information  associated  with 
the  measurement.  Frequencies  of  the  various  measurements 
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in  the  database  were  determined,  both  in  aggregate  and 
by  measurement  type.  The  categorization  of  participants 
by  experience  level  and  whether  they  had  previously 
served  onboard  aegis  ships  was  used  in  presenting  the 
results.  However,  due  to  small  sample  size  and  no 
noticeable  differences  between  categories  all  subsequent 
analysis  was  performed  on  all  participants  as  a  single 
group . 
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IV.  USABILITY  EVALUATION  RESULTS 


A.  BACKGROUND 

The  results  of  this  usability  evaluation  are 
presented  in  the  same  order  they  were  collected.  The 
participant's  initial  impressions  of  the  DSS-2  graphical 
user  interface  are  presented  along  with  the  participants 
initial  impressions  of  the  six  major  components.  The 
DSS-2  components  include  the  Figures  2  through  7 ; 


Figure  2 :  Toolbar 


Figure  3 :  Geoplot 
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Figure  6 :  Response  Manager 


Figure  7 :  Track  Suniaary 
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The  participant's  task  completion  rates,  post-task 
question  responses,  and  the  answers  to  the  post-test 
questionnaire  are  also  provide. 

B.  INITIAL  IMPRESSIONS 

Overall,  participant's  first  impressions  of  the  DSS 
were  positive  (Table  5)  .  Participants  generally  found 
the  DSS  to  be  a  familiar  interface  that  contained  more 
information  than  they  were  accustomed  to  in  existing 
shipboard  systems.  In  addition,  participants  stated  that 
their  initial  impression  of  the  DSS-2  was  that  it  aided 
situational  awareness  and  is  used  to  consolidate 
information.  In  particular,  participants  were  familiar 
with  the  geoplot  map  display  and  understood  that  the 
toolbar  would  be  used  for  manipulating  the  geoplot  map 
(Tables  6  and  7).  Participants  generally  understood  that 
the  minicros  were  summaries  of  individual  track 
information  and  that  this  information  was  ordered  in 
some  manner  (Table  8) .  Three  participants  either  did  not 
know  what  the  minicros  would  be  used  for  or  incorrectly 
identified  the  meaning  of  the  minicro  functionality. 
Participants  had  difficulty  identifying  the  track 
profile  component  of  the  DSS-2  (Table  9)  .  A  majority  of 
participants  incorrectly  believed  the  Track  Profile 
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component  to  be  a  weapons  status  monitor  that  reflected 
the  status  and  quantities  of  weapons  available.  The 
response  manager  was  unfamiliar  to  all  participants 
(Table  10).  However,  the  concept  of  the  response  manager 
was  generally  understood  by  all  participants. 
Participants  stated  that  the  response  manager  would  deal 
with  engagement  orders,  doctrine,  rules  of  engagement, 
recommended  actions,  or  a  checklist.  Generally 
participants  correctly  identified  the  track  summary 
component,  however  three  participants  incorrectly 
assumed  that  the  track  summary  information  pertained  to 
own-ship  status  (Table  11)  . 
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Table  5:  What  is  your  first  impression  of  what  you  see? 


•  Familiar  Geoplot. 

•  Like  it,  layout.  Black  and  white  stand  out. 

•  Similar  to  JOTS  display. 

•  Where  we  are  and  where  our  battlegroup  is.  Focus  on 
situational  awareness,  where  we  are. 

•  Difficult. 

•  Situational  Awareness . 

•  A  lot  of  crap  on  the  screen,  cluttered,  overwhelming. 

•  Lot  of  information,  used  to  pick  out  symbols. 

•  Consolidated  a  lot  of  info. 

•  Immediately  obvious,  right  sight. 

•  Looks  cool.  Intimidating. 

•  Like  it,  used  to  one  screen.  Get  more  info  with  this. 


Table  6:  What  does  the  far  left  side  of  the  left  monitor 


represent  to  you? 

(Toolbar) 

• 

Control  panel  for  display  and  tracks . 

• 

Power  point. 

• 

View  of  situational  picture. 

• 

Track  contact  info,  select  track 
decisions  for  you. 

symbology,  make 

• 

Legend . 

• 

Filter  setting. 

• 

Legend  for  map. 

• 

NTDS  notations . 

• 

Toolbar  for  geoplot. 

• 

Alter  geoplot. 

• 

Select  what  you  want  to  look  at. 
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Table  7:  What  does  the  upper  left  area  of  the  left 
monitor  represent  to  you?  (Geoplot) 

•  Map. 

•  JOTS  display,  NTDS. 

•  Where  I  am  and  what  my  radar  knows. 

•  Visual  display  of  geographical  picture. 

•  Operating  area,  tactical  area. 

•  Geoplot . 

•  Northern  Persian  Gulf. 

•  Geoplot  of  surface  and  air  contacts. 

•  Threat  access,  radar  responsiveness  in  a  certain  area. 

•  Big  picture,  where  we  are. 

•  Topographical  map.  _ 


Table  8:  What  does  the  bottom  row  on  both  monitors 
represent  to  you?  (Minicros) 

•  Quick  Summary  to  what  you  are  seeing. 

•  Classification  of  targets,  not  sure  how  it  classifies. 

•  What  I  know  of  the  tracks  in  my  area. 

•  Contact  information,  track  #'s  and  names. 

•  Weapon  employment  areas . 

•  Contacts. 

•  Information  of  ships. 

•  Current  tracks .  Air  and  merchants  set  as  priority  time  or 
threat . 

•  Contact  bearing  range,  speed,  and  sensor  types. 

•  One  for  each  track. 

•  Nothing,  selection  buttons? _ 
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Table  9:  What  does  the  top  left  area  of  the  right 
monitor  represent  to  you?  (Track  profile) 

•  X  axis  questions  and  range. 

•  What  our  weapons  are. 

•  Status  of  weapons  systems . 

•  Weapons  status,  a  horizontal  bar  chart. 

•  TAO  stuff.  Order  of  steps,  things  to  do. 

•  Nothing,  weapons  status? 

•  My  weapons  and  how  much  I  have. 

•  Our  weapons . 

•  How  much  I  have,  status,  and  range. 

•  Not  quite  sure,  our  weapons  status? 

•  Weapons  status .  _ 


Table  10:  What  does  the  middle-left  area  of  the  right 
monitor  represent  to  you?  (Response  Manager) 


• 

Engagement  orders . 

• 

When  things  happen,  what  should  happen  at 
taken  place. 

COI/ 

already 

• 

Never  seen  anything  like  it.  A  decision  matrix 

for  ROE, 

possible  defenses,  weapons  posture.  Where 

we  are  and  what 

we  should  do . 

• 

Decision-making.  Decisions  I  need  to  make. 

• 

I  don't  know,  maybe  doctrine  statement. 

• 

Doctrine . 

• 

Distance  time  line,  envelopes.  Need  to  do 

something. 

• 

ROE ' s . 

• 

Priority  of  actions.  Recommended  actions  by  system. 

• 

Don't  know,  a  continuiun  of  todo's? 

• 

Time  line. 
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Table  11:  What  does  the  top  right  area  of  the  right 
monitor  represent  to  you?  (Track  Summary) 

•  Track  details . 

•  Close  control,  more  information,  emitter  age. 

•  Contact  COI . 

•  Has  a  tracks  detailed  information. 

•  Specific  track  information.  Don't  know  how  it  is 
different  from  (minicros) ,  maybe  more  specific. 

•  All  I  need  to  know  to  launch. 

•  Own  ship,  don't  like  presentation. 

•  Ship  status . 

•  Contact  information. 

•  Blow  up  of  minitrack  status.  Active/intel/last  known 
position. 

•  Our  own  ship  status. 

•  Ships  information  and  dependent  on  current  selection. 


Once  data  was  gathered  on  the  participants  initial 
impressions  of  the  DSS  and  its  components,  a  series  of 
tasks  were  presented.  Participants  completed  the  tasks 
and  answered  post-task  questions  concerning  ease-of-use 
and  satisfaction.  Task  completion  errors  were  recorded 
and  task  completion  percentages  were  calculated.  Task 
completion  rates  which  were  below  the  usability 
criterion  of  90%  are  highlighted  in  Table  12  and 
discussed  in  Chapter  five. 
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C.  TASK  COMPLETION 


Table  12 :  Task  Errors  and  Completion  Rate 


TASK 

Errors 

Completion 

Rate 

Task  1:  Display  the  track  numbers  of  all  contacts  in 
the  map  display. 

4 

Task  2:  Locate  and  select  track  number  7012. 

0 

100% 

Task  3 :  Change  the  map  to  display  the  directions  all 
tracks  are  moving. 

1 

92% 

Task  4:  Remove  all  unknown  tracks  from  the  map 
display. 

0 

100% 

Task  5:  Display  all  surface  unknown  tracks. 

3 

rT5% 

Task  6:  Display  all  tracks. 

0 

100% 

Task  7:  Change  the  size  of  the  map  to  better  see  the 
tracks  displayed. 

3 

15% 

Task  8:  Identify  the  range  of  map  sizes  that  are 
available. 

12 

11 

Task  9:'  Explore  these  two  buttons.  When  finished 
exploring,  select  map  size  of  128  nm. 

0 

100% 

Task  10:  Please  point  to  own-ship  on  the  map  display. 

1 

92% 

Task  11:  Identify  which  tracks  are  potential  threats. 

6 

Task  12:  Determine  whether  track  7013  is  within  its 
weapons  release  range. 

3 

m 

Task  13:  Determine  whether  track  7013  is  within  own- 
ship's  weapons  envelope. 

1 

92% 

Task  14:  Identify  the  bearing  and  range  of  track 
7016. 

0 

100% 

Task  15:  Check  and  see  what  the  most  recent  warning 
information  is  regarding  track  7011. 

.  3 

fill 

Task  16:  Identify  the  most  recent  warning  information 
for  track  7011. 

2 

§3% 

Task  17:  Select  track  7016. 

0 

100% 

Task  18:  Locate  own-ship  position  on  the  right 
monitor . 

2 

»3% 

Task  19:  Using  information  available  on  the  right 
monitor,  determine  whether  track  7016  is  within  its 
weapons  release  range. 

3 

Task  20:  Using  information  available  on  the  right 
monitor,  determine  whether  track  7016  is  within  own- 
ship's  5/54  guns  weapons  envelope. 

0 

100% 

Task  21:  Select  track  7017. 

0 

100% 

Task  22:  Click  on  verify  airspace,  issue  a  level  1, 
and  change  CIWS  to  auto/ready. 

1 

92% 

Task  23:  Select  track  7012. 

0 

100% 
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Figures  8  through  26  show  participant  responses  to 
each  of  the  post-task  questions.  Each  figure  shows 
either  ease-of-use  or  satisfaction  ratings  for  all  12 
participants.  The  bars  in  the  figures  represent 
individual  participant  ratings.  These  bars  are  grouped 
according  to  the  category  of  user  and  the  bars  within 
each  grouping  are  order  according  to  relative  experience 
levels  within  the  group.  Tables  13  through  43  summarize 
participant  comments  and  what  participants  thought  could 
make  the  completion  of  the  task  easier. 
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•  Had  to  look  for  it.  Training. 

•  Clutters  display,  lose  visual  reference  of  contacts. 

•  Just  a  little  more  time  to  study  screen. 

•  Very  easy  considering  1st  time  use.  Change  buttons  to 
read  easier  (e.g.)  "Display  all  track  #'s". 

•  Shift  click  or  mousedrag  over  all  contacts. 

•  Different  color  background  to  make  it  standout  add  to  the 
"show"  title  for  instance  "Display  on  Map". 

•  I  was  not  sure  what  I  was  doing  was  going  to  work  the 
first  time.  Experience  with  the  interface. 

•  Took  a  step  and  had  to  decide  if  it  was  a  collective 
action  or  serial.  Pre-knowledge. 

•  Easy  once  I  knew  what  to  do. 

•  Advanced  training,  change  label  to  "show  all  tracks"  or 

something  like  that.  Experience. _ 
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Lower  Non-Aegis  Upper  Non-Aegis  Lower  Aegis  Upper  Aegis 

Participants 


Figure  9:  How  easy/difficult  was  it  to  find  track  number 

7012?  (Question  2) 

Table  14:  Comments  &  what  would  make  this  easier? 

(Question  2} 

•  Put  track  #'s  in  order  on  bottom  display. 

•  What  will  happen  to  display  w/in  traffic  zones,  >20 
contacts . 

•  No  problem. 

•  Need  a  10  key  to  enter  track  # . 

•  Experience  with  the  interface. 

•  Track  ntomber  hides  symbol.  If  the  tracks  exceed  the 
display.  Area,  may  need  a  summairy  list  that  is 
categorical . 

•  If  I  understood  the  ordering  of  the  track  #'s  at  the 
bottom. 
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Figure  10:  How  easy/difficult  was  it  to  read  the  track 
number  on  the  map  display?  (Question  3) 

Table  15:  Comments  &  what  would  make  this  easier? 

(Question  3) 

Possibility  of  having  #’s  separate.  This  has  its 
disadvantages  too ! 

On  large  scale  chart  hard  to  distinguish  individual  # ’ s 
Make  larger. 

Looking  for  track  #  highlighted.  Arrange  tabs  by  other 
'sorts ' . 

Flashing  contact  #  or  something  to  make  it  stand  out  -  gets 
larger  for  example. 

How  far  from  screen  will  the  user  be?  More  than  one  user 
may  have  to  use  a  display. _ 
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Participants 

Figure  11:  How  easy/difficult  was  it  to  display  the 
course  leaders  in  the  map  display?  (Question  4) 

Table  16:  Comments  &  what  would  make  this  easier? 

(Question  4) 

•  Used  history  function  first. 

•  Display  all  buttons. 

•  Experience . 

•  Experience. _ 
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Participants 


Figure  12:  How  easy/difficult  was  it  to  remove  unknown 
tracks  from  the  map  display?  (Question  5) 


Table  17 :  Comments  &  what  would  make  this  easier? 

(Question  5) 


• 

Had  to  click  to  determine  if  on/off. 

• 

Did  it  individually  by  air,  sfc,  sub,  not  in  one  step. 

• 

Little  time  needed  to  understand  there  were 

toggle 

buttons . 

• 

Very  easy  since  I  had  the  experience  of  the 

previous 

task . 

• 

Experience . 

53 


7  -| 

f  6- 

i2 

£ 

*o 
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Figure  13:  How  easy/difficult  was  it  to  display  all 
surface  unknown  tracks  on  the  map  display?  (Question  6) 

Table  18 :  Comments  &  what  would  make  this  easier? 

(Question  6) 

•  Symbology  is  a  plus . _ 
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Participants 


Figure  14:  How  easy/difficult  was  It  to  understand  when 
the  track  type  buttons  were  selected?  (Question  7) 


Table  19 :  Comments  &  what  would  make  this  easier? 

(Question  7} 

•  Training . 

•  Noise  was  good  indication,  but  won't  work  in  shipboard 

environment.  Distinct  color  change  would  be  better,  (e.g. 
black  vs.  white).  _ ^ _ • 
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does? 


Question  8:  How  would  you  explain  what  this 


Table  20:  Comments  &  what  would  make  this  easier? 

(Question  8) 


• 

Zooms  by  factor  2 . 

• 

Goes  down  to  next  pre-selected  button  and  increases  by 

1/2  the  current  range. 

• 

Decreases  range  scale. 

• 

Decreases  map  size,  smaller 

range  scale. 

• 

"+"  =  Zoom  In. 

• 

Zoom  in. 

• 

Zoom  in. 

• 

Zooms  out,  lowers  scale. 

Zooms  in  range  on  increment 

per  click. 

• 

Increase  zoom. 

• 

Zooms  in  range  on  increment 

per  click. 

• 

Zoom  in. 

Question  9. 


How  would  you  explain  what  this 


Table  21:  Comments  &  what  would  make  this  easier? 

(Question  9) 

•  Unzooms  by  factor  2 . 

•  Decreases  by  2  the  current  range. 

•  Increases  range  scale. 

•  Increases  map  size,  larger  range  scale. 

•  "-"=Zoom  Out. 

•  Zoom  out . 

•  Zoom  out . 

•  Zooms  in . 

•  Zooms  out  range  on  increment  per  slide. 

•  Decrease  zoom,  increase  size  of  area  covered. 

•  Zooms  out . 

•  Zooms  out. 
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Question  10.  Does  the  positioning  of  these  buttons 

(plus  on  the  left  /  minus  on  the  right)  meet  your 

expectations?  Five  participants  responded  yes,  seven 

subjects  said  no. 

Table  22:  Comments  &  what  would  make  this  easier? 

(Question  10} 

•  Numbers  decrease  to  the  left,  therefore  sign  should  be  to 
the  left. 

•  Expect  minus  on  left  associate  small  w/left,  large 
w/right. 

•  At  first  I  had  them  reversed. 

•  "+"  on  the  Right? 

•  I  expected  the  normal  to  the  left  though  I  saw  after 

trial  that  it  means  to  increase  the  scale.  For  me  it  was 
a  compatibility  error. 

•  Would  help  to  have  a  drag  zoom  to  center  and  zoom. 

•  Seems  like  "+"  should  be  on  the  right,  on  the  left. 

•  Prefer  to  be  able  to  select  my  own  choice  of 

magnification. _  _ _ 
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Upper  Non-Aegis  Lower  Aegis 

Participants 


Figure  15:  How  easy/difficult:  was  it  to  identify  own- 
ship  on  the  map  display?  (Question  11) 

Table  23:  Comments  &  what  would  make  this  easier? 

(Question  11) 

•  Standard  NTDS  symbology  used. 

•  Hook  self. 

•  Provide  legend  in  Track  types  for  own  ship  or  the 
different  nodes  show  that  this  is  a  geo-center  display. 

•  I  assumed  that  the  viewer  could  always  choose  the  center 
of  the  screen  wherever  he  wants . 
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Upper  Aegis 


Lower  Non-Aegis  Upper  Non-Aegis  Lower  Aegis 

Participants 

Figure  16:  How  easy/difficult  was  it  to  identify  whch 
tracks  were  potential  threats?  (Question  12) 

Table  24:  Comments  &  what  would  make  this  easier? 

(Question  12) 

•  Standard  NTDS  Threats  are  red,  unknown  white,  etc. 

•  Color  on  model  is  yellow/ expected  NTDS  standard  of  red. 
Either  use  NTDS  colors  and  symbols  or  more  time  and 
familiarity  with  model. 

•  Color  is  deceiving,  I  was  drawn  to  colored  icons,  not  all 
threats  a  TGT  that  had  not  had  any  additional  evaluation, 
would  not  be  colored,  and  could  be  missed.  What  is  a 
threat  in  this  scenario,  can  I  change  that  criteria  and 
then  display  them? 

•  This  is  ambiguous  at  best.  In  Gulf  all  tracks  are 

potential  threats . _ 


Lower  Non-Aegis 


Upper  Aegis 


Upper  Non-Aegis  Lower  Aegis 

Participants 

Figure  17 :  How  easy/difficult  was  it  to  determine 
whether  track  7013  was  within  its  weapons  release  range? 

(Question  13) 

Table  25:  Comments  &  what  would  make  this  easier? 

(Question  13) 

•  Comparing  to  left  screen  made  it  clearer. 

•  Multiple  envelopes  are  going  to  be  hard  to  see. 

•  This  information  is  only  based  on  an  ass\amed 
configuration,  may  lead  to  wrong  decision  making. 

•  Not  sure  what  the  red  and  gray  grids  mean.  This  is  really 
just  a  training  issue,  but  easily  learned. 

•  Experience  with  the  interface. 

•  Pre  knowledge  of  red  versus  white  would  make  it  easier. 

•  Difficult  to  figure  it  out  the  first  time.  Rename  button 

to  say  "weapons  ranges". _ 
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Lower  Non-Aegis  Upper  Non-Aegis  Lower  Aegis  Upper  Aegis 

Participants 


Figure  18:  How  easy/difficult  was  it  to  determine 
whether  track  7013  was  within  own-ship's  weapons  release 

range?  (Question  14) 


Table  26:  Comments  &  what  would  make  this  easier? 

(Question  14) 


• 

Comparing  to  left  screen  made  it  clearer. 

• 

Multiple  envelopes  are  going  to  be  hard  to 

see . 

• 

This  is  known  info,  easy  to  decide. 

• 

This  is  really  just  a  training  issue  but  easily 

learned. 

• 

Because  I  was  guided  to  the  weapons  button 

on  the 

left 

screen. 

• 

Pre  knowledge  of  red  versus  white  would  make  it 

easier. 

• 

Difficult  to  figure  it  out  the  first  time. 

Rename 

button 

to  say  "weapons  ranges". 

• 

Experience . 
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Participants 

Figure  19:  How  easy/difficult  was  it  to  identify  the 
bearing  and  range  of  track  7016?  (Question  15) 


Table  27 :  Comments  &  what  would  make  this  easier? 

(Question  15) 

•  Familiarity. 

•  Bottom  tab  not  so  clear.  Bearing  =??,  Range=??. 

•  I  need  compass  display,  not  for  this,  but  relative 
positions  for  ship's  head  for  weapons  envelopes. 

•  Not  readily  recognizable,  but  it  is  only  due  to 
familiarity  with  the  display. 

•  Titles  for  course  and  range  displayed. 

•  Unsure  whether  312/16.6  represents  range/brg  or  cse/sro. 
Label  the  names . 
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Question  16.  Do  you  think  there  is  any  meaning  to 
the  ordering  of  these  items?  Eleven  participants  said 
yes,  one  said  no. 

Table  28:  If  yes,  what  do  you  think  the  ordering  means? 

(Question  16) 

•  Range  and  threat. 

•  Threat . 

•  Threat  level .  Unsure  of  ordering  within  categories 
(Threat,  Unk,  Nuet)  no  apparent  categorization  by 
platform.  Appears  to  list  closest  contact  first. 

•  Organized  from  higher  potential  threats  to  lower 
potential  threats . 

•  Yes,  excellent  quick  reference  for  track  data.  Ordering 
according  to  threat? 

•  Don't  always  know  what  is  driving  the  ordering. 

•  Higher  threats  on  left. 

•  Threat  priority. 

•  Potential  threat  order. 

•  I  think  they  are  ordered  in  te3nns  of  threat . 

•  Ordered  by  threat /unk/ friend  and  then  by  contact  order. 
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Figure  20:  How  easy/difficult  was  it  to  identify  the 
most  recent  warning  information  for  track  7011? 

(Question  17) 

Table  29:  Comments  &  what  would  make  this  easier? 

(Question  17) 

•  Familiarity  and  knowing  "alerts"  had  to  be  clicked  on  the 
symbol . 

•  Training . 

•  Unfamiliar  with  pull  down  menu.  Not  require  continued 
user  interface  with  mouse  to  view  alert  menu. 

•  My  Understanding  of  Question?  Button  to  read  "Warnings" 

•  Track  updated,  I  did  not  see  and  called  wrong 
information! ! ! 

•  What  are  those  numbers,  if  its  time  in  needs  to  be  in 
00:00:00  format . 

•  Easily  found  and  understood. 

•  Except  I  had  to  hold  the  mouse  button  down  to  view 
alerts.  Menu  should  stay  up  after  clicked. 

•  Pre  knowledge . 

•  Didn't  know  where  to  find  it.  Maybe  a  flashing  warning 
light . 

•  Experience. _ 
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Figure  21:  How  easy/difficult  was  it  for  you  to  select 
the  alert  button  and  view  the  alert  window?  (Question 

18) 


Table  30:  Comments  &  what  would  make  this  easier? 

(Question  18) 

•  Familiarity. 

•  Unfamiliar  with  pull  down  menu.  Not  require  continued 
user  interface  with  mouse  to  view  alert  menu. 

•  Didn't  catch  my  eye  right  away. 

•  Couple  times  to  learn  to  hold  down. 

•  Kind  of  small.  Keep  alert  up,  let  any  button  close 
window . 

•  Easily  found  and  understood. 

•  Except  I  had  to  hold  the  mouse  button  down  to  view 
alerts.  Menu  should  stay  up  after  clicked. 

•  I  would  like  it  to  stay  in  view  when  clicked. _ 
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Question  19:  Click  and  hold  the  alert  button.  What 


do  the  numbers  on  the  right  mean  to  you?  Seven 

participants  believe  the  number  on  the  right 

corresponded  to  the  time  elapsed  since  warning  occurred 

and  one  the  time  the  warning  occurred. 

Table  31:  Comments  &  what  would  make  this  easier? 

(Question  19) 

•  Blue  is  significant,  gray  contact  updates. 

•  Lt.  Blue  means  increase  threat  issue. 

•  White:  general  info.  Blue:  ROE  specific  info. 

•  Green  appears  to  be  a  more  serious  alert. 

•  Blue=Warning  or  threat,  Grey=neutral  info. 

•  Blue  is  associated  attack/defending. 

•  Color  means  new  alert,  use  red. 

•  Blue  means  what  affect  if  has  on  me.  Grey  is  general 
information. 

•  Contact  within  weapons  range,  I  would  expect  that  the 
colors  would  change  (maybe  to  red)  as  the  contact  gets 
closer. 

•  White  general,  blue  warning,  red?  Perhaps  hostile  action. 

•  Blue  means  high  significance. 

•  White-narrative,  Blue-threat. 
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Figure  22:  How  easy/difficult  was  it  to  understand  what 
the  numbers  in  the  alert  window  mean?  (Question  20) 


Table  32:  Comments  &  what  would  make  this  easier? 

(Question  20) 


• 

I  just  didn't  know  when  I  saw 
new  to  me. 

them,  time  since 

report 

is 

• 

Training . 

• 

Still  unsure.  Familiarity  with  system  or  standard  use 

of 

Zulu  time  on  all  areas . 

• 

Colxomn  header  =  Time  Elapsed. 

• 

Both  actual  and  elapsed  time  on  target. 

• 

00:00:00  format. 

• 

Use  a  plus  symbol  next  to  the 

time  to  indicate 

how  long 

since  time  zero . 

• 

No  indication  of  what  they  mean.  Labeling. 
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Figure  23:  How  easy/difficult  was  it  to  understand  what 
the  colors  in  the  alert  window  mean?  (Question  21) 

Table  33 :  Comments  &  what  would  make  this  easier? 

(Question  21) 

•  I  am  still  not  sure.  Familiarity,  if  I  used  the  system 
the  colors  would  be  easy  to  remember. 

•  Add  a  feature  for  contacts  weapons  release  range  in  RED. 

•  Need  to  find  pattern.  Use  more  than  2  color  if  it  is 
going  to  colorize. 

•  Use  red. 

•  .1  had  to  think  about  it. 

•  Yellow  is  a  better  warning  color,  blue  is  too  passive. 

•  No  indication  of  what  they  mean.  Labeling. _ 
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Participants 

Figure  24:  How  easy/difficult  was  it  to  determine 
whether  track  7016  was  within  its  weapons  release  range 
using  the  infromation  displayed  in  the  right  monitor? 

(Question  22) 

Table  34 :  Comments  &  what  would  make  this  easier? 

(Question  22) 

•  Familiarity. 

•  Training . 

•  I  need  a  representation  giving  weapon's  masking  areas 
unless  I  know  the  graphs  account  for  it. 

•  History  is  less  important  than  current  velocity  for 
weapons . 

•  I'm  not  sure  how  to  interpret  the  graph.  Training. _ 
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Figure  25:  How  easy/difficult  was  it  to  determine 

whether  track  7016  was  within  own-ship's  weapons  release 
range  using  the  information  displayed  in  the  right 
monitor?  (Question  23) 

Table  35:  Comments  &  what  would  make  this  easier? 

(Question  23) 

•  Training-but  by  now  I  have  some. 

•  Have  available  weapons  as  a  pull  down  menu.  Click 
weapons,  release  mouse,  see  choices,  make  choice  by 
clicking  desired  weapon. 

•  Select  and  drag  is  hard,  select/select . 

•  Easily  understood. 

•  Make  weapons  window  a  pop  down  and  stay  vice  hold  down 
and  find. 

•  I'm  not  sure  how  to  interpret  the  graph.  Training. _ 
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Table  36:  What  do  you  expect  the  ASPECT  button  to  do? 

(Question  24} 


•  Show  angle  on  bow  for  TGT  and  TGT  angle 

•  Show  highlighted  COI  target  angle,  heading,  altitude,  etc 

•  I  don ' t  know 

•  2=D  shift  by  90  degrees  or  180  degrees.  3-D  view 
possibly? 

•  TGT  aspect 

•  Target  w/relation  to  ship's  head. 

•  Change  from  center  on  me  to  a  center  on  him. 

•  Place  threat  contact  at  (0,0)  axis. 

•  Weapons  release  envelope  of  current  velocity. of  contact. 

•  I  don ' t  know 

•  Target  aspect. _ 


Question  25:  The  aspect  button  only  met  one 

participants  expectations,  ten  said  that  it  did  not. 

Table  37 :  Comments  &  what  would  make  this  easier? 

(Question  25) 

•  It  tells  own  ship  aspect  to  TGT,  which  is  easy,  and  now  I 
expected  target  angle. 

•  It's  a  good  tool,  but  w/ "aspect",  I  expected  to  see  "nose 
on"  "crossing",  etc. 

•  It  give  whether  your  weapons  engagement  criteria  are  met 
or  not . 

•  Took  me  a  few  seconds  to  understand  what  it  was  telling 
me.  Also,  the  weapons  button  is  hard  to  use. 

•  What's  it  mean? 

•  Needs  to  be  a  lot  bigger,  should  be  scalable  like  map, 
should  be  primary  weapons  display. 

•  No  reason,  I’m  just  off. 

•  I  did  think  that  it  would  do  what  it  did  until  I  tried 
it,  and  then  it  made  sense. 

•  It  is  a  weapons  engagement  aspect  for  own  ships  weapons . 

•  Because  I  didn't  know  what  it  was  for. 

•  Does  not  give  me  target  aspect,  provides  weapons  I  can 

use  against  target  with  my  present  heading. _ 
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Figure  26:  How  satisfied/dissatisfied  are  you  with  the 
ASPECT  button  feature?  (Question  26) 

Table  38:  Coxoments  &  what  would  make  this  easier? 

(Question  26) 

•  I  like  it! 

•  Satisfied  with  the  information  it  is  giving  me,  not  with 
its  ease-of-use. 

•  Excellent!  Rename  button  to  "Arc  of  Coverage". 

•  Bad  button  name,  it  is  showing  weapon  cut  outs,  we  all 
worry  where  we  can  not  shoot  and  how  to  maneuver  to 
shoot.  Change  color  if  in  and  out,  edge  discrimination. 

•  Once  I  understood  what  it  meant,  this  feature  is 
excellent . 

•  Needs  to  be  in  a  bigger  window  and  maybe  a  different 
name. 

•  Should  be  bigger. 

•  Not  what  I  expected. 


Table  39:  What  would  you  expect  to  do  with  this  area 
based  on  the  track  selected?  (Question  27) 

•  Use  it  for  reports  and  ROE 

•  Go  down  the  check  list  to  see  what ' s  been  done 

•  Time  line  for  deconf liction,  verification  of  COI,  weapons 
status  upgrade,  countermeasures  and  reports.  Basically 
how  to  classify  or  fight  this  target  according  to  ROE. 

•  Interact /warning 

•  Line  coincides  with  doctrine  packages  (I.e.  follow  these 
doctrine) 

•  Checklist. 

•  Conduct  doctrine  based  query  factors . 

•  Perform  the  actions  that  the  vertical  line  is  in. 

•  I  think  that  I  can  click  on  what  warnings  I  have  given 
and  the  display  will  change  colors  to  remind  me  later. 

•  Respond  to  recommendations . 

•  Do  the  action  when  the  line  reaches  the  left  side  of  the 
bubble . 

•  Actions  can  take  at  this  range.  _ 


Table  40:  What  did  clicking  in  the  Response  Manager  mean 

to  you?  (Question  28) 

•  Orders  were  given  to  do  the  three  task  clicked 

•  Only  that  the  task  has  been  completed  by  someone  else 

•  Actions  have  been  taken. 

•  Issued  track  a  warning,  took  a  self-defense  measure  with 
CIWS,  checked  for  territorial  airspace. 

•  Told  CIC  that  were  in  level  1  &  switch  to  CIWS  auto. 

•  It  has  been  done. 

•  Should  be  being  carried  out. 

•  I  am  verifying  that  my  airspace  is  clear,  issuing  a  level 
1  warning  to  the  contact  and  setting  CIWS  into  auto. 

•  Memory  aid  not  a  command  to  change  CIWS  status  a  display 
of  current  status . 

•  Click:  issue  order  to  appropriate  station. 

•  No  idea. 

•  When  within  range  the  selected  actions  will  take  place. 
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Table  41:  What  does  the  greyed-out  area,  change  CIWS  to 
auto/ready  mean  to  you?  (Question  29) 

•  Already  ordered. 

•  Only  that  the  TAO  thinks  it's  done. 

•  DSS  is  recommending  this  course  of  action. 

•  Means  the  system  is  standing  by  to  fire  on  a  potential 
threat . 

•  That  doctrine  is  being  followed  -active. 

•  That  is  a  ship  system,  it  is  already  in  that  state  or 
been  clicked. 

•  Means  it  is  already  complete  w/regard  to  7013. 

•  The  CIWS  will  automatically  track  and  the  system  is 
operational . 

•  Means  that  my  CIWS  status  is  auto/ready. 

•  The  CIWS  is  still  in  auto/ready. 

•  It's  been  accomplished. 

•  Places  CIWS  in  auto/ready. _ 

Table  42:  What  is  the  relationship  between  the  Track 
Profile  and  the  Response  Manager?  (Question  31) 

•  They  are  on  same  scale 

•  Range  of  contact,  engagability  vs.  defensive  steps  taken 
by  own  ship 

•  Range  of  COI  is  the  same  top  shows  physical  engagement 
limits  bottom  shows  ROE  and  weapons,  posture 
recommendations . 

•  One  displays  possible  decisions  to  make  on  a  track  and 
the  other  provides  a  visible  aspect  of  that  decision. 

•  Top-gives  threat  envelopes  and  the  track  bottom  gives 
what  doctrine  to  follow  according  to  the  track. 

•  Show  points  in  relative  distance  that  require /suggest  or 
move  points  of  action. 

•  Distance  axis. 

•  Spatial  in  that  as  the  contact  further  progresses  in  the 
envelope  certain  tasks  must  be  completed. 

•  They  show  that  a  contact  is  within  my  weapons  range  and 
tracks  the  ROE  that  I  have  followed. 

•  Range  of  contact  to  ship.  Range  of  Recommended  actions. 

•  They  are  covered  by  range  of  target  to  own  ship. 
Represents  the  actions  that  need  to  be  taken. 

•  Both  display  range  to  target  ship. 
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Table  43 :  What  does  the  Response  Manager  white  line  mean 

to  you?  (Question  32) 

•  Actual  range. 

•  Range  to  own  ship. 

•  COI  range. 

•  Range  of  contact  to  ship,  also  gives  an  indication  of 
where  you  should  be  in  your  decision  making  process. 

•  The  doctrine  to  follow  according  to  that  specific  track. 

•  Present  distance. 

•  Distance  target  is  from  ship. 

•  Distance  line. 

•  Decreasing  range  of  contact. 

•  Range  of  contact  in  question. 

•  Range  to  target  ship. _ _ 

Figure  27  shows  the  post-task  rating  average  for 
each  of  the  participants.  For  example,  the  bar  farthest 
to  the  left  of  Figure  27  shows  that  the  average  rating 
given  across  all  post-task  questions  by  this  participant 
to  be  3.0.  Figure  28  shows  the  average  post- task  rating 
for  each  post-task  question.  For  example,  the  bottom  bar 
representing  question  one  indicates  that  the  average 
rating  for  this  question  was  approximately  2.5. 
Questions  for  which  the  rating  exceeded  the  usability 
criterion  of  3.0  include  questions: 

•  13  -  How  easy/difficult  was  it  to  determine 

whether  track  7013  was  within  its  weapons  release 
range . 

•  17  -  How  easy/difficult  was  it  to  'identify  the 

most  recent  warning  information  for  track  7011? 

•  20  -  How  easy/difficult  was  it  to  understand  what 
the  numbers  in  the  alert  window  mean? 

•  21  -  How  easy/difficult  was  it  to  understand  what 
the  colors  in  the  alert  window  mean? 
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Immediately  following  the  usability  evaluation 
participants  were  given  a  post-test  questionnaire. 
Figures  29  through  42  show  participant  ratings  for  ease- 
of-use  and  satisfaction  for  the  DSS  and  its'  components. 
All  ratings  were  within  the  established  usability 
criteria  with  the  exception  of  the  Track  Profile.  The 
Track  Profile  component  received  an  average  ease-of-use 
rating  of  3.2  and  participant  satisfaction  rating  of 
3.2. 
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Figure  29:  How  easy/difficult  was  it  for  you  to  use  the 

Geoplot?  (Question  1) 

Table  44:  How  could  the  Geoplot  be  improved? 

•  Own  ship  speed  leader. 

•  Switch  +/-  on  range  scales. 

•  Good  geoplot,  detailed  digital  maps  would  be  excellent. 

•  Declutter  button  like  the  one  on  the  JMCIS.  Add  symbology 
to  the  plot . 

•  Zoom  area  option.  Center  ship  option.  Speed  leader  on  own 
ship . 

•  Be  able  to  choose  center  instead  of  always  own-ship. _ 
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Figure  30:  How  satisified/dissatisfied  were  you  with  the 

Geoplot?  (Question  2) 
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Figure  31:  How  easy/difficult  was  it  for  you  to  use  the 

Minicros?  (Question  3) 

Table  45:  How  could  the  Minicros  be  improved? 

•  Limit  information  on  Minicro  so  more  can  be  displayed. 

•  Hold  training  on  capabilities. 

•  Use  labels  for  course,  speed,  range,  altitude,  and 
associated  units. 

•  Did  the  bearing  and  range  represent  CPA  data  or  the 
contacts  bearing  and  range? 

•  Threats  need  more  markers,  anything  w/closing  CPA  needs 
to  be  marked  in  some  manner. 

•  What  happens  if  list  is  too  much? 

•  Labeling  of  Range/Dist  or  CSE/Speed. 

•  Display  track  course  and  speed. _ 
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Figure  32:  How  satisfied/dissatisfied  were  you  with  the 

Minicros?  (Question  4) 
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Figure  33:  How  easy/difficult  was  it  for  you  to  use  the 
Track  Proflile?  (Question  5) 

•  Training. 

•  Make  own  ships  weapons  a  pull-down  menu  w/o  continuous 
user  control  of  mouse. 

•  Aspect  and  weapon  buttons  were  a  little  difficult  to 
understand  at  first. 

•  The  "aspect"  picture  seems  to  be  the  better  one  to  use. 
Suggest  switching  the  two  pictures. 

•  Need  weapons  envelopes  on  top  down  like  aspect.  Needs  to 
be  primary  interface. 

•  Remove  history,  add  speed  leader.  Use  different  color  for 
history  and  speed  leader. 

•  Unsure  how  to  interpret.  Training. 

•  Too  many  choices.  All  weapons  on  1  display. _ 
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Figure  34:  How  satisified/dissatisf ied  were  you  with  the 

Track  Profile?  (Question  6) 


80 


7 


I  6- 
£ 


Lower  Non-Aegis  NotvAegis  Lower  Aegis  Upper  Aegis 

Participants 


Figure  35:  How  easy/difficult:  was  it  for  you  to  use  the 

Toolbar?  (Question  7) 

Table  46:  How  could  the  Toolbar  be  improved? 

•  Complete  NTDS  symbology. 

•  Needed  training. 

•  Make  threat  color  red. 

•  +  and  -  range  buttons . 

•  Change  clicks  into  distinct  visible  color  changes. 

•  Add  a  select  all,  remove  the  sound  when  selecting. 

•  Be  able  to  combine  background  elements . 

•  Make  it  so  I  can  choose  my  own  magnification. _ 
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Figure  36:  How  satisified/dissatisfied  were  you  with  the 

Toolbar?  (Question  8) 
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Figure  37:  How  easy/difficult  was  it  for  you  to  use  the 
Response  Manager?  (Question  9) 

•  Tell  me  if  these  are  directives  or  recommendations. 

•  Excellent  doctrine  tool . 

•  I  think  feedback  on  time  action  complete  could  be  added 
to  each  line,  so  TAO  knows  it's  done. 

•  If  distance  line  passes  into  the  area,  have  the  area 
color  stand  out  until  task  is  completed. 

•  Unsure  what  it  is  used  for.  Training. 
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Figure  38:  How  satisif ied/dissatisfied  were  you  with  the 
Response  Manager?  (Question  10) 
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Figure  39:  How  easy/difficult  was  it  for  you  to  use  the 
Track  Summary?  (Question  11) 

•  Training . 

•  Allow  user  to  manually  type  in  track  # .  Need  to  add  time 
to  CPA  and/or  time  to  COI  at  weapons  release  range. 

•  CPA  was  a  little  difficult  to  catch  at  first. 

•  I  had  trouble  finding  the  CPA. 

•  Track  selection  is  hard  to  discover  needs  marker,  so  you 
can  tell  it's  a  button. 

Add  for  keyboard  numerical  pad  change  of  contact.  Further 
explain  CPA  by  adding  CPA  bearing  and  CPA  range  lines . 
Clicking  and  holding  on  the  track  #  to  display 
info/select  a  new  track  did  not  seem  natural  to  me. 
Increase  size  of  own  ship  CRS  and  SPD.  Add  own  ship  speed 
leader . 
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Figure  40:  How  satisfied/dissatisfied  were  you  with  the 
Track  Summary?  (Question  12) 
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Figure  41:  Overall,  how  easy/difficult  was  it  for  you  to 

use  the  DSS?  (Question  13) 


•  Training . 

•  Pull-down  menus  for  alerts  and  own  ships  weapons. 

•  Training,  more  use  of  color  or  shape  as  indicators, 
rather  than  sound. 

•  Very  high  learning  curve.  Vast  improvement  to  tactical 
display  and  management. 

•  Ensure  track  number  does  not  block  symbology. _ 

3.5-1 
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Figure  42:  Overall,  how  satisfied/dissatisfied  were  you 
with  the  DSS?  (Question  14) 


Table  47 :  What  changes  would  you  like  to  make  to  the 

DSS?  (Question  15) 

•  Faster  response  to  "weapon"  (upper  right)  and  have  an 
alert  to  the  right  of  minicro  that  says  "more  contacts". 

•  Can  you  modify  the  Response  Manager  for  changing  ROE, 
optasks,  or  missions  such  as  amphibious  assault  or 
maritime  interdiction  ops? 

•  +  and  -  on  range  scales  reversed. 

•  Colors  are  "eye-catchers"  and  even  have  meanings.  Use 
very  descriptive  words.  Change  track  profile  picture  to 
the  "aspect"  picture  that  was  clearer  to  me.  Place  track 
data  in  table  format  (this  may  save  space  and  be  easier 
to  look  at) . 

•  Change  weapon  envelopes  into  geographic  aspect  display  to 
account  for  weapon's  masking. 

•  Allow  keypad  input  for  the  contact  number  in  Track 
Summary.  Add  titles  to  CPA  BRNG  and  CPA  RNGE  and  add  an 
estimated  time  to  CPA.  Add  a  plus  or  minus  to  the  alert 
times,  not  how  long  after  the  alert  was  posted. 

•  Slight  interface  change.  For  example,  have  the  menu  stay 
up  after  you  click  the  alerts  for  a  contact. 


Table  48:  What  were  the  worst  aspects  of  the  DSS 
interface  and  why?  (Question  16) 

•  Split  screen. 

•  No  time  to  CPA.  No  time  to  COI  w/in  its  weapons  release 
range  or  w/in  your  weapons  release  range. 

•  Increase  the  size  of  the  "CPA"  indicator.  Weapon  pull¬ 
down  menu  is  difficult  to  use. 

•  Track  profile  picture  is  still  a  little  confusing.  Keep 
things  simple. 

•  No  idea  about  some  features  because  they  give  no 
indications  they  are  buttons . 

•  Too  much  reliance  on  the  mouse.  With  2  screens  and 
looking  in  other  areas  besides  the  2  monitors,  the  mouse 
arrow  is  easily  lost. 

•  Weapons  button  was  difficult  to  find  and  understand. 

•  History  in  weapons  display,  aspect  display. _ 
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Table  49:  What  were  the  best  aspects  of  the  interface 

and  why?  (Question  17) 

•  Ease-of-use. 

•  Good  data,  lots  of  info  displayed  here. 

•  Easy  to  see  threats  and  weapons  envelopes. 

•  The  decision  making  process  time  line  with  target  range. 

•  Use  of  colors.  All  on  "1"  screen.  It  was  all  right  there. 

•  Geographic  w/map  display  is  nice. 

•  Minicro-I  like  the  quick  and  pertinent  information  on  the 
contact.  Response  Manager-excellent  doctrine  aide. 
Aspect-allows  for  quick  maneuvering  recommendations  to 
OOD. 

•  Overall  I  think  it  was  an  easy  to  use  interface. 

•  Liked  having  so  much  information  simultaneously  visible. 

Table  50:  Where  there  any  parts  of  the  interface  that 
you  found  confusing  or  difficult  to  understand? 

(Question  18) 

•  Multiple  threat  displays  possibly  pop-up,  sizeable 
windows . 

•  No,  not  really. 

•  Not  really,  difficulties  primarily  due  to  unfamiliarity 
w/system. 

•  It  took  me  a  few  minutes  to  understand  aspect. 

•  Track  profile  picture  -  it  showed  weapon  engagement 
envelopes  but  I'm  still  a  little  confused  by  it.  The 
"aspect"  picture  was  clearer  to  me.  I'm  not  sure  what  the 
track  profile  picture  axes  were? 

•  Time  late  on  the  alert  box  needs  to  be  00:00:00  format. 

•  Adding  new  contact  as  far  as  reporting  one  that  is  not  on 
the  minicro.  It  is  easy  to  get  used  to  the  display  and 
not  look  at  the  geoplot . 

•  Weapons  button.  Track  summary. 

•  Track  profile  area . 
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V.  SUMMARY,  CONCLUSIONS,  AND  RECOMMENDATIONS 

A.  SUMMARY 

The  DSS  did  not  meet  these  usability  objectives  of 
90%  across  all  tasks  for  task  completion.  The  overall 
task  completion  rate  across  all  tasks  was  84%,  6%  below 

the  usability  criterion  initially  established.  When 
examined  by  individual  task,  on  13  tasks  the  DSS 
surpassed  the  90%  usability  criterion  level  and  on  10 
tasks  the  DSS-2  did  not  meet  the  usability  objective. 
The  only  task  on  which  all  participants  committed  an 
error  was  the  identification  of  the  map's  upper  and 
lower  range  scale.  Half  of  the  participants  were  unable 
to  properly  identify  potential  threats  and  one-third  of 
the  participants  had  difficulty  displaying  all  the  track 
numbers  on  the  map  display.  A  quarter  of  all 
participants  committed  errors  on  five  different  tasks. 
These  tasks  were: 

•  Display  all  surface  unknown  tracks; 

•  Change  the  size  of  the  map  to  better  see  the 
tracks  displayed; 

•  Check  what  the  most  recent  warning  information  is 
regarding  track  7011; 

•  Determine  whether  track  7013  is  within  its 

weapons  release  range; 

•  Determine  whether  track  7016  was  within  its 

weapons  release  range  using  the  track  profile 
component . 
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Two  participants  incorrectly  identified  the  most  recent 
warning  information  for  track  7011. 

Overall,  the  DSS-2  did  not  meet  the  90%  usability 
objective  across  all  tasks  for  ease-of-use.  Eighty-two 
percent  (8%  below  the  90%  objective)  of  the  ease-of-use 
questions  averaged  a  rating  of  somewhat  easy  or  better. 
The  four  questions  that  did  not  meet  the  ease-of-use 
usability  objective  goals  were: 

•  How  easy/difficult  was  it  to  determine  whether 
track  7013  was  within  its  weapons  release  range? 

•  How  easy/dif  f  icult  was  it  to  identify  the  most 
recent  warning  information  for  track  7011? 

•  How  easy/difficult  was  it  to  understand  what  the 
numbers  in  the  alert  window  mean? 

•  How  easy/dif f icult  was  it  to  understand  what  the 
colors  in  the  alert  window  mean? 

Four  ease-of-use  questions  received  a  rating  of  six  or 

higher  by  at  least  on  participant,  where  a  rating  of 

five  corresponded  to  somewhat  difficult  and  a  rating  of 

seven  corresponded  to  difficult .  These  questions  were: 

•  How  easy/difficult  was  it  to  display  all  tracks? 

•  How  easy/dif f icult  was  it  to  identify  whether  track 
7013  was  within  its  weapons  release  range? 

•  How  easy/difficult  was  it  to  understand  what  the 
numbers  in  the  alert  window  mean? 

•  How  easy/difficult  was  it  to  determine  whether 
track  7016  was  within  it's  weapons  release  range 
using  the  information  displayed  in  the  right 
monitor? 
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The  usability  objective  for  satisfaction  was  met,  100% 
of  satisfaction  ratings  were  below  a  rating  somewhat 
satisfied . 

During  the  usability  study,  participants  were 
questioned  about  the  functionality  and  user  interface 
design  of  some  components  of  the  DSS-2 .  The  following 
information  was  compiled  from  participant'  responses. 

The  range  scale  feature  presented  some  difficulty 
to  participants.  Participants  were  able  to  correctly 
explain  how  the  range  magnification  buttons  functioned 
after  exploring  the  button.  However,  seven  participants 
stated  that  they  believed  that  the  labeling  of  the 
buttons  was  reversed. 

When  questioned  about  the  ordering  of  the  minicros, 
all  but  one  participant  believed  the  minicros  were 
ordered  in  some  manner.  Of  the  eleven  participants  who 
believed  the  minicros  were  ordered,  nine  correctly 
assumed  that  they  were  ordered  by  threat.  When  asked  to 
select  a  track  that  did  not  have  a  corresponding 
minicro,  four  of  the  12  participants  committed  an  error. 
A  majority  of  participants  expressed  surprise  that  some 
tracks  did  not  have  a  corresponding  minicro. 

All  12  participants  had  difficulty  selecting  the 
alert  button  on  the  minicro.  In  addition,  although 
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eleven  participants  correctly  identified  the  numbers 
found  in  the  alert  window,  every  participant  expressed 
frustration  that  these  numbers  were  not  labeled.  Four 
participants  stated  that  additional  or  different  formats 
for  these  numbers  might  be  more  appropriate.  In  general, 
participants  expressed  significant  dislike  for  the  way 
the  alert  buttons  had  to  be  selected.  The  interaction  of 
having  to  click  and  hold  the  button  to  read  the 
information  contained  within  the  pop-up  window  caused 
frustration.  Many  participants  felt  that  this 
interaction  would  be  inappropriate  during  shipboard 
operations . 

The  aspect  button  surprised  almost  all 
participants.  Ten  participants  stated  that  this  button 
did  not  meet  their  expectations,  however,  nearly  all 
participants  liked  the  feature.  The  predominate 
criticism  of  the  aspect  button  was  its  size,  many 
participants  expressed  that  it  was  too  small. 

The  post-test  questionnaire  examined  the 
participants  overall  experience  with  the  DSS-2  and  with 
each  individual  component.  Overall,  results  of  the  post- 


test  questionnaire 

suggest 

that  the 

DSS-2  met 

the 

usability 

obj  active 

criteria 

initially 

established 

of 

somewhat 

easy  or 

better 

for  ease-of-use 

and 
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satisfaction.  The  average  DSS-2  ease-of-use  rating  was 
2.6  and  the  average  satisfaction  rating  was  2.3.  The 
only  DSS-2  component  that  did  not  meet  the  ease-of-use 
and  satisfaction  usability  objectives  in  the  post-test 
questionnaire  was  the  track  profile  component.  This 
component  received  a  3.2  for  ease-of-use  and 
satisfaction.  All  participants  reported  that  they 
enjoyed  working  with  the  DSS-2  and  believed  that  it 
would  be  a  significant  step  forward  in  CIC  information 
management . 

B .  RECOMMENDATIONS 

Table  51  details  the  usability  issues  encountered 
by  study  participants  during  this  usability  evaluation 
and  provides  corresponding  recommendations. 


Table  51:  DSS  Component  Issues  and  Recommendations 


COMPONENT 

ISSUE 

RECOMMENDAT I ON 

Geoplot 

Track  symbology 

1.  Some  participants  did  not  like 
the  coloring  of  potential 
threats  and  unknown  tracks . 
Participants  stated  that 
unknowns  were  potential 
threats . 

1 .  Conduct  further 
research  to 
determine  the 
optimal  use  of 
track  color. 

Toolbar 

Track  Numbers 

1.  Some  participants  wanted  the 
track  number  button's  name  to 
be  changed  to  "Display  all 
tracks  # ' s" . 

1.  Change  existing 
button  label. 
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Range  Magnification 

1.  All  participants  had 

difficulty  discovering  the 
upper  and  lower  bounds  of  the 
range  scale. 

2  ,  Many  participants  chose  the 

incorrect  range  magnification 
button  to  resize  the  geoplot. 

3  .  Some  participants  wanted  to  be 

able  to  enter  any  range  using 
a  numerical  keypad. 

4.  One  participant  wanted  to  be 
able  to  use  the  mouse  on  the 
geoplot  to  select  an  area  to 
be  magnified  by  clicking  and 
dragging  across  the  area. 

1.  Remove  button 
option  and 
replace  with  a 
slider  widget. 

2  -  Change  the  range 
magnification 
icons  relative 
positions . 

3 .  Provide  numerical 
keypad  capability 
to  enter  a 
specific  range 
magnification . 

4.  Provide  this 
dynamic  map  range 
selection 
capability . 

Minicro 

General 

1.  Many  participants  wanted 

1.  Provide  more 

abbreviated  labels  on  the 

labeling  within 

information  displayed. 

the  alert  window. 

2 .  Several  participants  were 

2 .  Maintain  the 

surprised  when  the  selected 

selected  state  of 

minicro  changed.  A  few  users 

the  minicro,  even 

looked  at  another  area  of  the 

if  a  priority 

DSS-2  and  did  not  expect  the 

changes . 

selection  to  change  when  the 

POTENTIAL 

priority  did. 

DANGEROUS . 

Alert  Information 

1.  Many  participants  expressed 

1.  Redesign  the 

frustration  when  they  could 

alert  selection 

not  bring  up  the  alert 

mechanism  so 

quickly.  Participants  clicked 

that  one  click 

on  the  alert  button  and  didn't 

will  open  alert 

realize  they  had  to  click  and 

window  and  a 

hold  it  to  bring  up  alert 

second  click 

window. 

will  close  it. 

2 .  Many  participants  wanted  the 

2 .  Label 

information  contained  in  the 

information. 

alert  window  to  be  labeled. 

3 .  Some  participants  wanted  the 

3 .  Test  the 

time  of  the  alert,  as  well  as 

feasibility  of 

the  time  since  the  alert,  to 

having  both 

be  displayed. 

options . 

4.  Many  participants  did  not  like 

4.  Additional 

the  light  blue  color  and 

research  on 

preferred  for  red  or  yellow. 

color . 
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Track 

Weapons  Envelope 

Profile 

1.  Some  participants  had 

1 .  Provide  training 

difficulty  identifying  which 

and  easily 

weapons  envelope,  red  or  blue. 

accessible  help 

related  to  the  selected  track. 

feature . 

2.  One  participant  wanted  the 

2 .  Provide  the 

ability  to  have  multiple 

capability  to 

weapons  envelopes  displayed 

have  multiple 

simultaneously. 

weapons 
envelopes . 

3.  One  participant  had  difficulty 
identifying  the  location  of 
own- ship . 

Aspect  Insert 

3.  Training. 

1.  Many  participants  stated  that 

1.  Redesign  and  test 

the  aspect  button  should  be 

alternative 

- 

labeled  "Weapons  Cut-Out"  or 
"Cut-Out" . 

labeling . 

2.  Many  participants  wanted  the 

2.  Increase  size  of 

Aspect  Insert  to  be  larger. 

Aspect  Insert - 

3.  Some  participants  preferred 
the  Aspect  Insert  to  the  Track 
Profile . 

3  .  Training . 

Response 

General 

Manager 

1.  Many  participants  stated  that 
they  were  unsure  whether  the 
options  in  the  Response 

Manager  were  recommendations 
or  requirements. 

1.  Training. 

2 .  Many  participants  wanted  to  be 

2 .  Existing  feature 

able  to  edit  the  Response 

that  was  not 

Manager . 

tested. 

Track 

General 

Summary 

1.  Some  participants  had 

1.  Redesign  and  test 

difficulty  reading  the 

alternative 

information  contained  in  the 

colors  to 

Track  Summary  due  to  poor 

facilitate 

background  and  text  coloring. 

reading . 

C.  CONCLUSIONS 

This  study  examined  the  usability  of  the  DSS-2  in  a 

systematic  manner  and  established  where  the  DSS-2 

today  against  usability 
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product  is 


criteria . 


Furthermore,  participants  identified  usability  issues 
that  can  be  addressed  in  future  design  and  research 
efforts.  Based  on  the  information  obtained  from  this 
evaluation,  recommendations  to  address  the  usability 
issues  were  made.  Overall,  in  terms  of  ease-of-use  and 
satisfaction,  participants  reacted  positively  to  the 
DSS-2 .  Participants  felt  the  DSS-2  would  assist  them  in 
maintaining  situational  awareness  and  was  a  tool  that 
would  be  useful  onboard  Navy  ships. 

The  methodology  applied  in  this  study  was  useful  in 
the  evaluation  of  the  DSS-2 .  This  study  demonstrated 
that  traditional  human- computer  interface  usability 
methods  could  be  directly  applied  the  evaluation  of 
synthetic  environments.  The  DSS-2  is  a  simple  synthetic 
environment  represented  on  two  computer  monitors .  Given 
the  success  of  this  methodology  with  the  DSS-2,  it  would 
be  appropriate  to  use  this  methodology  in  evaluating 
more  complex  synthetic  environments . 
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APPENDIX  A:  CONSENT  FORM 
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CONSENT  FORM 


Usability  Evaluation  of  the  Decision  Support  System 

Principal  Investigator:  LT  Dylan  Schmorrow 

Operations  Research  Department 
Naval  Postgraduate  School 
Monterey,  CA  93943 

I, _ ,  consent  to  my  participation  in  the  research  project  titled 

Usability  Evaluation  of  the  Decision  Support  System. 

I  understand  that  I  am  free  to  withdraw  my  participation  in  the  research  at  any  time  and  that  if  I  do 
I  will  not  be  subjected  to  any  penalty  or  discriminatory  treatment. 

I  have  been  given  the  opportunity  to  ask  questions  about  the  research  and  received  satisfactory 
answers. 

I  understand  that  any  information  or  personal  details  gathered  in  the  course  of  this  research  about 
me  are  confidential  and  that  neither  my  name  nor  any  other  identifying  information  will  be  used  or 
published  without  my  written  permission. 

I  understand  that  if  I  have  any  complaints  or  concerns  about  this  research  I  can  contact: 

George  Conner 

Operations  Research  Department 
408-656-3306 


Signed  by: 


Date 


APPENDIX  B 


:  DEMOGRAPHIC  INFORMATION  FORM 
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DEMOGRAPHIC  INFORMATION 


Name:  _  Rank:  _  Designator:  _ 

Years  of  service:  _  Time  in  rank  (months):  _  Curriculum:  _ 

Previous  Command:  _ _  Primary  billet:  _ 

Months  onboard:  _  Months  in  shipyard  (if  applicable):  _ 

Months  standing  CIC  watch: _  Months  standing  TAO  watch:  _ 

Hours  spent  on  computer  each  week: _ Operating  system  primarily  used: 

Qualifications: 

CIC  Watch  Officer? 

SWOS  Department  Head  Course? 

SWC  Qualified? 

STWO  Qualified? 

TAO  Qualified? 

Aegis  Qualified? 

Other? _ 


Experience  Summary  --  List  assignments  relevant  to  SWO  experience 
(include  Combat  Systems) 


Command  Months 


Circle  One  Date 

Yes  No  _ 

Yes  No  _ 

Yes  No  _ 

Yes  No  _ 

Yes  No  _ 

Yes  No 


LANT 

PAC 

MED 

PERGULF 

Others 


APPENDIX  C:  TASK  SCRIPT 
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Background 

This  project  was  spawned  by  the  1988  USS  Vincennes 
incident  where  an  Aegis  cruiser  engaged  in  a  littoral 
warfare  peace-keeping  mission  shot  down  an  Iranian 
Airbus.  Investigations  following  the  incident  suggested 
that  stress  may  have  affects  on  decision  making,  and 
that  these  effects  were  not  well  understood.  This 
project  was  established  to  address  these  concerns. 

This  prototype  Decision  Support  System  (DSS)  was 
developed  to  enhance  Navy  tactical  decision  making  based 
on  "naturalistic"  decision  processes.  Displays  were 
developed  to  support  critical  decision  making  tasks  by 
Naval  watch  officers  operating  in  a  shipboard  Combat 
Information  Center. 

SCENARIO 

This  scenario  has  the  ship  operating  independently 
in  the  northern  Persian  Gulf  50  nm  to  the  east  of  Kuwait 
City.  You  are  on  a  presence  patrol  and  have  been 
directed  to  remain  within  5  nm  of  your  current  position 
to  demonstrate  US  resolve.  Weapons  and  warning  status: 
Yellow  and  Tight.  At  scenario  start  you  are  on  course 
020,  speed  7  knots.  Visibility  is  reduced  to 
approximately  4  to  5  nm  in  dust  and  haze.  Local  time  is 
1100  . 
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Task  1 


Display  the  track  numbers  of  all  contacts  in  the  map 
display. 


Task  2 

Locate  and  select  track  number  7012. 
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Post  Task  Questions 


1.  How  easy/difficult  was  it  to  display  all  the  track 
numbers  ? 


1  2  3  4  5  6  7 

easy  somewhat  somewhat  difficult 

easy  difficult 

Comments : _ 


What  might  have  made  this  task  easier?. 


2.  How  easy/difficult  was  it  to  find  track  n'umber  7012? 


1  2  3  4  5  6  7 

easy  somewhat  somewhat  difficult 

easy  difficult 

Comments: _  _ 


What  might  have  made  this  task  easier?. 


3 .  How  easy/difficult  was  it  to  read  the  track  number  on 
the  map  display? 

_ I _ I _ I - 1 - 1 

1  2  3  4  5  6 

easy  somewhat  somewhat 

easy  difficult 

Comments : _ 

What  might  have  made  this  task  easier? _ 


7 

difficult 


Task  3 

Change  the  map  to  display  the  directions  all  tracks  are  moving. 


Task  4 

Remove  all  unknown  tracks  from  the  map  display. 


Task  5 

Display  all  surface  unknown  tracks . 


Task  6 

Display  all  tracks . 
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Post  Task  Questions 


4.  How  easy/difficult  was  it  to  display  the  course  leaders 
in  the  map  display? 

I _ I _ I _ I _ I - 1 - 1 

1  2  3  4  5  6  7 

easy  somewhat  somewhat  difficult 

easy  difficult 

Comments : _ 


What  might  have  made  this  task  easier?, 


5.  How  easy/difficult  was  it  to  remove  unknown  tracks  from 
the  map  display? 

I _ I _ I _ I _ I - 1 - 1 

1  2  3  4  5  6  7 

easy  somewhat  somewhat  difficult 

easy  difficult 

Comments  _ _ _ 


What  might  have  made  this  task  easier?. 


6.  How  easy/difficult  was  it  to  display  all  surface  unknown 
tracks  on  the  map  display? 


1  2  3  4  5  6  7 

easy  somewhat  somewhat  difficult 

easy  difficult 

Comments : _ 


What  might  have  made  this  task  easier?, 


7.  How  easy/difficult  was  it  to  understand  when  the  track 
type  buttons  (i.e.,  surface,  unknowns)  were  selected. 


1  2  3  4  5  6  7 

easy  somewhat  somewhat  difficult 

easy  difficult 

Comments : _ 


What  might  have  made  this  task  easier?. 


Task  7 


Change  the  size  of  the  map  to  better  see  the  tracks 
displayed. 


Task  8 

Identify  the  range  of  map  sizes  that  are  available. 
Read  aloud  the  upper  and  lower  range  levels. 


Task  9 

Explore  these  two  buttons.  When  finished  exploring,  select 
map  size  of  128  nm. 


Post  Task  Questions 


8 .  How  would  you  explain  what 


this  button  does 


9  .  How'  would  you 


explain  what  this  button  does 


10.  Does  the  positioning  of  these  buttons  (plus  on  the 
left  /  minus  on  the  right,  meet  your  expectations? 

Yes  /  No 


If  no,  why  not?, 
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Task  10 


Please  point  to  own-ship  on  the  map  display. 


Task  11 

Identify  which  tracks  are  potential  threats? 


Task  12 

Determine  whether  track  7013  is  within  its  weapons  release 
range . 


Task  13 

Determine  whether  track  7013  is  within  own-ship's  weapons 
envelope . 
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Post  Task  Questions 


11.  How  easy/difficult  was  it  to  identify  own-ship  on  the 
map  display. 

I _ I _ I _ I _ I _ I 

1  2  3  4  5  6 

easy  somewhat  somewhat 

easy  difficult 

Comments : _ 


What  might  have  made  this  task  easier?. 


12.  How  easy/difficult  was  it  to  identify  which  tracks 
were  potential  threats? 


1  2  3  4  5  6 

easy  somewhat  somewhat 

easy  difficult 

Comments : _ 


What  might  have  made  this  task  easier?. 


7 

difficult 


7 

difficult 


i 


13.  How  easy/difficult  was  it  to  determine  whether  track 
7013  was  within  its  weapons  release  range?* 


1  2  3  4  5  6  7 

easy  somewhat  somewhat  difficult 

easy  difficult 

Comments  : _ ^ _ 


What  might  have  made  this  task  easier?, 


14.  How  easy/difficult  was  it  to  determine  whether  track 
7013  was  within  own-ship's  weapons  release  range? 

I _ I _ I _ I _ I _ I _ I 

1  2  3  4  5  6  7 

easy  somewhat  somewhat  difficult 

easy  difficult 

Comments : _ 


What  might  have  made  this  task  easier?. 


r 


Task  14 

r 

Identify  and  read  aloud  the  bearing  and  range  of  track 
7016. 
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Post  Task  Questions 


15.  How  easy/dif ficult  was  it  to  identify  the  bearing  and 
range  of  track  7016? 


1  2 
easy 


3  4 

somewhat 
easy 


5  6 

somewhat 
difficult 


7 

difficult 


Comments : 


What  might  have  made  this  task  easier? 


16 .  Do  you  think  there  is  any  meaning  to  the  ordering  of 
these  items? 

YES  /  NO 

If  yes,  what  do  you  think  the  ordering  means? 
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Task  15 


The  DSS  is  continually  updating  information  on  the  behavior 
of  all  tracks .  Check  and  see  what  the  most  recent  warning 
information  is  regarding  track  7011. 


Task  16 

Identify  and  read  aloud  the  most  recent  warning  information 
for  track  7011. 
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Post  Task  Questions 


17.  How  easy/difficult  was  it  to  identify  the  most  recent 
warning  information  for  track  7011? 


12  3  4 

easy  somewhat 

easy 


5  6  7 

somewhat  difficult 

difficult 


Comments : _ 

What  might  have  made  this  task  easier?. 


18.  How  easy/difficult  was  it  for  you  to  select  the  alert 
button  and  view  the  alert  window? 


12  3  4 

easy  somewhat 

easy 


5  6  7 

somewhat  difficult 

difficult 


Comments : _ 

What  might  have  made  this  task  easier?, 


19 .  Click  and  hold  one  of  the  alert  buttons . 

A.  What  do  the  numbers  on  the  right  mean  to  you? 

_  Time  elapsed  since  warning  occurred. 

_ _  Time  warning  occurred. 

_  Other,  please  described 

B.  What  do  the  colors  mean  to  you? 
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20.  How  easy/difficult  was  it  to  understand  what  the 
numbers  in  the  alert  window  mean? 


1  2  3  4  5  6  7 

easy  somewhat  somewhat  difficult 

easy  difficult 

Comments : _ 


What  might  have  made  this  task  easier? 


21.  How  easy/difficult  was  it  to  understand 

what 

the 

colors  in  the  alert  window  mean? 

I  I  1  1  1 

1 

1 

1  2  3  4  5 

easy  somewhat  somewhat 

easy  difficult 

6 

7 

difficult 

Comments : 

What  miaht  have  made  this  task  easier? 

Task  17 


Select  track  7016. 


Task  18 

Locate  own-ship  symbology  on  the  right  monitor. 


TASK  19 

Using  information  available  on  the  right  monitor,  determine 
whether  track  7016  is  within  its  weapons  release  range . 


TASK  20 

Using  information  available  on  the  right  monitor,  dete2rmine 
whether  track  7016  is  within  own-ship's  5/54  Guns  weapons 
envelope . 
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Post  Task  Questions 


22 .  How  easy/difficult  was  it  to  determine  whether  track 
7016  was  within  its  weapons  release  range  using  the 
information  displayed  in  the  right  monitor? 

I _ I _ I _ I _ I _ I 

1  2  3  4  5  6 

easy  somewhat  somewhat 

easy  difficult 

Comments : _ 


What  might  have  made  this  task  easier? 


23 .  How  easy/difficult  was  it  to  determine  whether  track 
7016  was  within  own-ship's  weapons  release  range  using 
the  infoimiation  displayed  in  the  right  monitor? 

- 1 _ I _ I _ I _ 1 

1  2  3  4  5  6 

easy  somewhat  somewhat 

easy  difficult 

Comments : _ 


What  might  have  made  this  task  easier?, 


7 

difficult 


7 

difficult 


24.  What  do  you  expect  the  ASPECT  button  to  do? 

25.  Click  it.  Does  this  meet  your  expectations? 
Yes/no 

If  no,  why  not?  _ _ _ _ 


26.  How  satisfied/dissatisfied  are  you  with  the  ASPECT 
button  feature? 


5  6  7 

somewhat  dissatisfied 

dissatisfied 


1 

satisfied 


2 


3 

somewhat 

satisfied 


4 


Post  Task  Questions 


Task  21 


Select  track  7017. 


Question  27 :  What  would  you  expect  to  do  with  this  area 
based  on  the  track  selected? 


Task  22 

Question  28:  Click  on  verify  airspace,  issue  a  level  1,  and 
change  CIWS  to  auto/ready.  What  does  it  mean  to  you? 


Task  23 

Select  track  7013 

Question  29:  Please  describe  what  the  dark  gray  area,  "CIWS 
to  auto/ready",  means  to  you. 


Did  you  expect  the  "CIWS  to  auto  ready"  to  be  dark  gray? 
YES  /  NO 

Are  the  actions  listed  here  REQUIRED  or  RECOMMENDED? 
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Fir«  Flnraa/^antlju^Sliiit " 


'  Engag#;/.  Do^ihjat^^  jiigi^’ ■jgjfeg: 


1  jRe^rte^tt 

I  “ 

Question  31:  What  is  the  relationship  between  the  two  areas 
above? 


Question  32:  What  does  the  white  line  represent  to  you? 
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APPENDIX  D:  POST-TEST  QUESTIONS 
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OVERALL  POST-TEST  QUESTIONS 


Figure  43:  Geo-Plot  with  Desaturated  Map  and  Variable 

Coded  Symbology 

1.  How  easy/difficult  was  it  for  you  to  use  the  Geo-Plot? 

I - 1 - 1 - 1 - 1 _ I _ I 

1  2  3  4  5  6  7 

easy  somewhat  somewhat  difficult 

easy  difficult 

Comments : _ _ _ 

How  could  this  area  be  improved? _ 

2 .  How  satisfied/dissatisfied  were  you  with  the  Geo-Plot? 

I - 1 - 1 - 1 _ I _ I _ I 

1  2  3  4  5  6  7 

satisfied  somewhat  somewhat  dissatisfied 

satisfied  dissatisfied 

Comments : _ _ 
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Figure  44:  Sample  Mini-CRO 


3.  How  easy/difficult  was  it  for  you  to  use  the  Minicros? 


12  3  4 

easy  somewhat 

easy 


5  6  7 

somewhat  difficult 

difficult 


Comments : _ 

How  could  this  area  be  improved?. 


4.  How  satisfied/dissatisfied  were  you  with  the  Minicros? 


1  2 
satisfied 


3  4 

somewhat 
satisfied 


5  6  7 

somewhat  dissatisfied 
dissatisfied 


Comments : 
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Figure  45:  Track  Profile  with  Aspect  Inset 


5.  How  easy/difficult  was  it  for  you  to  use  the  Track 
Profile? 


1  2 
easy 


3  4 

somewhat 
easy 


5  6  7 

somewhat  difficult 

difficult 


Comments : 


How  could  this  area  be  improved? 


6.  How  satisfied/dissatisfied  were  you  with  the  Track 
Profile? 


1 

satisfied 


somewhat 

satisfied 


5  6  7 

somewhat  dissatisfied 
dissatisfied 


Comments : 


JSj-jck ground  . 


Course  Idrs  E}-i 


-Track  Types  - 


Air  I  ;  Sarf^oc  1 


uniBovai]  'ifil  r|ti| 
yrigBdgyj  : 


7.  How  easy/dif f icult  was  it  for  you  to  use  this  area? 


12  3  4 

easy  somewhat 

easy 


5  6  7 

somewhat  difficult 

difficult 


Comments : _ 

How  could  this  area  be  improved?. 


8.  How  satisfied/dissatisfied  were  you  with  this  area? 


1 

satisfied 


somewhat 

satisfied 


5  6  7 

somewhat  dissatisfied 

dissatisfied 


Comments : 
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Figure  46:  Response  Manager 


9.  How  easy/difficult  was  it  for  you  to  use  the  Response 
Manager? 


1  2 
easy 


3  4 

somewhat 
easy 


5  6  7 

somewhat  difficult 

difficult 


Comments : _ 

How  could  this  area  be  improved?. 


10.  How  satisfied/dissatisfied  were  you  with  the  Response 
Manager? 


1  2 
satisfied 


3  4 

somewhat 
satisfied 


5  6  7 

somewhat  dissatisfied 

dissatisfied 


Comments : 
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:  •  "~7  jr\  o 


i Super  Pumafielo 


Figure  47 :  Track  Summazry 


11.  How  easy/difficult  was  it  for  you  to  use  the  Track 
Summary? 


12  3  4 

easy  somewhat 

easy 


5  6  7 

somewhat  difficult 

difficult 


Comments : _ 

How  could  this  area  be  improved?. 


12.  How  satisfied/dissatisfied  were  you  with  the  Track 
Summary? 


1  2 
satisfied 


3  4 

somewhat 
satisfied 


5  6  7 

somewhat  dissatisfied 

dissatisfied 


Comments : 


129 


Figure  48:  TADMUS  DSS  Integrated  Display 


13.  Overall,  how  easy/difficult  was  it  for  you  to  use  the 
DSS? 


12  3  4 

easy  somewhat 

easy 


5  6  7 

somewhat  difficult 

difficult 


Comments : _ 

How  could  this  area  be  improved?. 


14.  Overall,  how  satisfied/dissatisfied  were  you  with  the 
DSS? 


1 

satisfied 


somewhat 

satisfied 


5  6  7 

somewhat  dissatisfied 

dissatisfied 


Comments : 
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Question  15 :  If  you  could  improve  the  DSS  system,  what 
changes  would  you  like  to  make? 


Question  16:  Overall  what  were  the  worst  aspects  of  the  DSS 
interface  and  why? 


Question  17 :  Overall  what  were  the  best  aspects  of  the  interface 
and  why?  Please  describe. 


Question  18:  Where  there  any  parts  of  the  interface  that 
you  found  confusing  or  difficult  to  understand.  Please 
Describe .  Please  describe . 
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